Dynamic Adaptation System for Surgical Simulation

ABSTRACT

Systems, methods, and instrumentalities for managing surgical consequences in a simulated surgical task may be provided. A simulation environment to simulate a surgical task defined by a set of environmental parameters may be received. Surgical interaction data may be received. In examples, the surgical interaction data may comprise an indication of a triggering event and/or an indication of consequence environmental parameters. The surgical interaction data may be associated with a surgical task. The consequence environmental parameters may be associated with the triggering event. User interaction data may be received. The user interaction data may be based on a user action within the simulation environment and may be associated with a surgical task. On a condition that the received user interaction data matches the triggering event, the simulation environment may be modified, for example, based on the consequence environmental parameters.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Provisional U.S. PatentApplication No. 63/191,681, May 21, 2021, the disclosure of which isincorporated herein by reference in its entirety.

This application is related to the following, filed contemporaneously,the contents of each of which are incorporated by reference herein:

-   -   U.S. patent application Ser. No. 17/332,594, filed May 27, 2021,        (Attorney Docket No. END9338USNP1), titled METHODS FOR SURGICAL        SIMULATION    -   U.S. patent application Ser. No. 17/332,524, filed May 27, 2021,        (Attorney Docket No. END9338USNP2), titled SURGICAL SIMULATION        OBJECT RECTIFICATION SYSTEM    -   U.S. patent application Ser. No. 17/332,399, filed May 27, 2021,        (Attorney Docket No. END9338USNP3), titled SURGICAL SIMULATION        NAVIGATION SYSTEM    -   U.S. patent application Ser. No. 17/332,441, filed May 27, 2021,        (Attorney Docket No. END9338USNP4), titled SURGICAL SIMULATION        SYSTEM WITH COORDINATED IMAGINING    -   U.S. patent application Ser. No. 17/332,462, filed May 27, 2021,        (Attorney Docket No. END9338USNP5), titled SURGICAL SIMULATION        SYSTEM WITH SIMULATED SURGICAL EQUIPMENT COORDINATION    -   U.S. patent application Ser. No. 17/332,197, filed May 27, 2021,        (Attorney Docket No. END9338USNP6), titled SIMULATION-BASED        SURGICAL PROCEDURE PLANNING SYSTEM    -   U.S. patent application Ser. No. 17/332,407, filed May 27, 2021,        (Attorney Docket No. END9338USNP7), titled SIMULATION-BASED        DIRECTED SURGICAL DEVELOPMENT SYSTEM    -   U.S. patent application Ser. No. 17/332,449, filed May 27, 2021,        (Attorney Docket No. END9338USNP8), titled SURGICAL ADVERSE        EVENT SIMULATION SYSTEM    -   U.S. patent application Ser. No. 17/332,496, filed May 27, 2021,        (Attorney Docket No. END9338USNP9), titled SIMULATION-BASED        SURGICAL ANALYSIS SYSTEM

BACKGROUND

Surgical simulations, such as computer-based, three-dimensionalsimulations of a surgical environment and/or surgical procedure forexample, present an opportunity to advance the surgical arts. Surgicalsimulations have potential to benefit surgical training, planning,development, and the like. For example, surgical simulations may be usedto train surgeons in new procedures and/or to improve the performance ofprocedures they already know. Surgical simulations may be used as avirtual “dress rehearsal” to help a surgeon prepare for an upcomingprocedure. And surgical simulations may be used to experiment withunproven procedures and techniques.

However, surgical simulation platforms are complex systems that facemany limitations in capabilities, scope, and applicability. For example,many platforms are technology “silos,” specifically programmed andtailored to address a particular learning objective or to simulate theoperation of a singular piece of equipment, such as simulating theoperation of a surgical robot. Limitations, such as these, may dimmish aplatform's effectiveness as a tool to advance the surgical arts. Andsuch limitations may represent significant technological roadblocks tothe integration of simulation-based applications into other aspects ofthe surgical process, such a pre-operative planning, intra-operativesupport, post-operative analysis, and the like.

Accordingly, innovation in surgical simulation technology, such astechnical advancements that address surgical simulation capabilities,scope, and applicability for example, may accelerate further progress inthe surgical arts.

SUMMARY

An interactive and dynamic surgical simulation system is disclosed. Thesurgical simulation system may be used in the context of acomputer-implemented interactive surgical system. For example, thesurgical simulation system may enable dynamic adaptation. For example,the surgical simulation system may provide rectification of surgicalsimulation objects. For example, the surgical simulation system mayenable enhanced navigation. For example, the surgical simulation systemmay provide coordinated surgical imagining. For example, the surgicalsimulation system may enable simulated surgical equipment coordination.For example, the surgical simulation system may provide simulation-basedsurgical procedure planning. For example, the surgical simulation systemmay enable simulation-based directed surgical development. For example,the surgical system may provide simulation of surgical adverse events.For example, the surgical system may enable simulation-based surgicalanalysis.

Systems, methods, and instrumentalities for managing surgicalconsequences in a simulated surgical task may be provided. A simulationenvironment to simulate a surgical task defined by a set ofenvironmental parameters may be received. Surgical interaction data maybe received. In examples, the surgical interaction data may comprise anindication of a triggering event and/or an indication of consequenceenvironmental parameters. The surgical interaction data may beassociated with a surgical task. In examples, the surgical task maycomprise a medical procedure and/or a medical procedure context. Theconsequence environmental parameters may be associated with thetriggering event. In examples, the triggering event and/or theconsequence environmental parameters may be based on historical dataassociated with the medical procedure and/or the medical procedurecontext. The historical data may be based on previously performed liveprocedures recorded by a surgical hub.

User interaction data may be received. The user interaction data may bebased on a user action within the simulation environment and may beassociated with a surgical task. On a condition that the received userinteraction data matches the triggering event, the simulationenvironment may be modified, for example, based on the consequenceenvironmental parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer-implemented interactive surgicalsystem.

FIG. 2 shows an example surgical system being used to perform a surgicalprocedure in an operating room.

FIG. 3 shows an example surgical hub paired with a visualization systema robotic system, and an intelligent instrument, in accordance with atleast one aspect of the present disclosure.

FIG. 4 illustrates a surgical data network having a communication hubconfigured to connect modular devices located in one or more operatingtheaters of a healthcare facility or any room in healthcare facilityspecially equipped for surgical operations to the cloud, in accordancewith at least one aspect of the present disclosure.

FIG. 5 illustrates an example computer-implemented interactive surgicalsystem.

FIG. 6 illustrates an example surgical hub comprising a plurality ofmodules coupled to the modular control tower.

FIG. 7. is a block diagram of an example surgical simulator system.

FIG. 8 is a block diagram of an example surgical simulator system.

FIG. 9 is a block diagram depicting an example surgical simulator userinterface device.

FIG. 10 is a flow chart of an example surgical simulator operation.

FIGS. 11A-B illustrate example surgical procedural plan data structuresfor use with a computer-implemented interactive surgical system and/or asurgical simulator.

FIG. 12 shows example surgical consequences management in a simulatedsurgical task.

FIG. 13 shows example surgical consequences management in a simulatedsurgical task.

FIG. 14 shows a layout of a simulator design for surgical consequencesmanagement in a simulated surgical task.

DETAILED DESCRIPTION

Surgical simulation systems, devices, and methods may include aspects ofintegration with other medical equipment, data sources, processes, andinstitutions. Surgical simulation systems, devices, and methods mayinclude aspects of integration with a computer-implemented interactivesurgical system and/or with one or more elements of acomputer-implemented interactive surgical system, for example.

Referring to FIG. 1, a computer-implemented interactive surgical system100 may include one or more surgical systems 102 and a cloud-basedsystem (e.g., the cloud 104 that may include a remote server 113 coupledto a storage device 105). Each surgical system 102 may include at leastone surgical hub 106 in communication with the cloud 104 that mayinclude a remote server 113.

One or more simulation devices 103, 111 may be in communication withand/or integrated as part of the computer-implemented interactivesurgical system 100. For example, the simulation device 103 may be anelement of the one or more surgical systems 102. For example, thesimulation device 103 may be in communication with one or more surgicalhubs 106. For example, the simulation device 111 may be in communicationwith the computer-implemented interactive surgical system 100 via thecloud 104.

In one example, as illustrated in FIG. 1, the surgical system 102includes a visualization system 108, a robotic system 110, and ahandheld intelligent surgical instrument 112, which are configured tocommunicate with one another and/or the hub 106. In some aspects, asurgical system 102 may include an M number of hubs 106, an N number ofvisualization systems 108, an O number of robotic systems 110, and a Pnumber of handheld intelligent surgical instruments 112, where M, N, O,and P may be integers greater than or equal to one.

In various aspects, the visualization system 108 may include one or moreimaging sensors, one or more image-processing units, one or more storagearrays, and one or more displays that are strategically arranged withrespect to the sterile field, as illustrated in FIG. 2. In one aspect,the visualization system 108 may include an interface for HL7, PACS, andEMR. Various components of the visualization system 108 are describedunder the heading “Advanced Imaging Acquisition Module” in U.S. PatentApplication Publication No. US 2019-0200844 A1 (U.S. patent applicationSer. No. 16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING,STORAGE AND DISPLAY, filed Dec. 4, 2018, the disclosure of which isherein incorporated by reference in its entirety.

As illustrated in FIG. 2, a primary display 119 is positioned in thesterile field to be visible to an operator at the operating table 114.In addition, a visualization tower 111 is positioned outside the sterilefield. The visualization tower 111 may include a first non-steriledisplay 107 and a second non-sterile display 109, which face away fromeach other. The visualization system 108, guided by the hub 106, isconfigured to utilize the displays 107, 109, and 119 to coordinateinformation flow to operators inside and outside the sterile field. Forexample, the hub 106 may cause the visualization system 108 to display asnapshot of a surgical site, as recorded by an imaging device 124, on anon-sterile display 107 or 109, while maintaining a live feed of thesurgical site on the primary display 119. The snapshot on thenon-sterile display 107 or 109 can permit a non-sterile operator toperform a diagnostic step relevant to the surgical procedure, forexample.

In one aspect, the hub 106 may also be configured to route a diagnosticinput or feedback entered by a non-sterile operator at the visualizationtower 111 to the primary display 119 within the sterile field, where itcan be viewed by a sterile operator at the operating table. In oneexample, the input can be in the form of a modification to the snapshotdisplayed on the non-sterile display 107 or 109, which can be routed tothe primary display 119 by the hub 106.

Referring to FIG. 2, a surgical instrument 112 is being used in thesurgical procedure as part of the surgical system 102. The hub 106 mayalso be configured to coordinate information flow to a display of thesurgical instrument 112. For example, in U.S. Patent ApplicationPublication No. US 2019-0200844 A1 (U.S. patent application Ser. No.16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY, filed Dec. 4, 2018, the disclosure of which is hereinincorporated by reference in its entirety. A diagnostic input orfeedback entered by a non-sterile operator at the visualization tower111 can be routed by the hub 106 to the surgical instrument display 115within the sterile field, where it can be viewed by the operator of thesurgical instrument 112. Example surgical instruments that are suitablefor use with the surgical system 102 are described under the heading“Surgical Instrument Hardware” and in U.S. Patent ApplicationPublication No. US 2019-0200844 A1 (U.S. patent application Ser. No.16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY, filed Dec. 4, 2018, the disclosure of which is hereinincorporated by reference in its entirety, for example.

FIG. 2 depicts an example of a surgical system 102 being used to performa surgical procedure on a patient who is lying down on an operatingtable 114 in a surgical operating room 116. A robotic system 110 may beused in the surgical procedure as a part of the surgical system 102. Therobotic system 110 may include a surgeon's console 118, a patient sidecart 120 (surgical robot), and a surgical robotic hub 122. The patientside cart 120 can manipulate at least one removably coupled surgicaltool 117 through a minimally invasive incision in the body of thepatient while the surgeon views the surgical site through the surgeon'sconsole 118. An image of the surgical site can be obtained by a medicalimaging device 124, which can be manipulated by the patient side cart120 to orient the imaging device 124. The robotic hub 122 can be used toprocess the images of the surgical site for subsequent display to thesurgeon through the surgeon's console 118.

Other types of robotic systems can be readily adapted for use with thesurgical system 102. Various examples of robotic systems and surgicaltools that are suitable for use with the present disclosure aredescribed in U.S. Patent Application Publication No. US 2019-0201137 A1(U.S. patent application Ser. No. 16/209,407), titled METHOD OF ROBOTICHUB COMMUNICATION, DETECTION, AND CONTROL, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.

Various examples of cloud-based analytics that are performed by thecloud 104, and are suitable for use with the present disclosure, aredescribed in U.S. Patent Application Publication No. US 2019-0206569 A1(U.S. patent application Ser. No. 16/209,403), titled METHOD OF CLOUDBASED DATA ANALYTICS FOR USE WITH THE HUB, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.

In various aspects, the imaging device 124 may include at least oneimage sensor and one or more optical components. Suitable image sensorsmay include, but are not limited to, Charge-Coupled Device (CCD) sensorsand Complementary Metal-Oxide Semiconductor (CMOS) sensors.

The optical components of the imaging device 124 may include one or moreillumination sources and/or one or more lenses. The one or moreillumination sources may be directed to illuminate portions of thesurgical field. The one or more image sensors may receive lightreflected or refracted from the surgical field, including lightreflected or refracted from tissue and/or surgical instruments.

The one or more illumination sources may be configured to radiateelectromagnetic energy in the visible spectrum as well as the invisiblespectrum. The visible spectrum, sometimes referred to as the opticalspectrum or luminous spectrum, is that portion of the electromagneticspectrum that is visible to (i.e., can be detected by) the human eye andmay be referred to as visible light or simply light. A typical human eyewill respond to wavelengths in air that are from about 380 nm to about750 nm.

The invisible spectrum (e.g., the non-luminous spectrum) is that portionof the electromagnetic spectrum that lies below and above the visiblespectrum (i.e., wavelengths below about 380 nm and above about 750 nm).The invisible spectrum is not detectable by the human eye. Wavelengthsgreater than about 750 nm are longer than the red visible spectrum, andthey become invisible infrared (IR), microwave, and radioelectromagnetic radiation. Wavelengths less than about 380 nm areshorter than the violet spectrum, and they become invisible ultraviolet,x-ray, and gamma ray electromagnetic radiation.

In various aspects, the imaging device 124 is configured for use in aminimally invasive procedure. Examples of imaging devices suitable foruse with the present disclosure include, but not limited to, anarthroscope, angioscope, bronchoscope, choledochoscope, colonoscope,cytoscope, duodenoscope, enteroscope, esophagogastro-duodenoscope(gastroscope), endoscope, laryngoscope, nasopharyngo-neproscope,sigmoidoscope, thoracoscope, and ureteroscope.

The imaging device may employ multi-spectrum monitoring to discriminatetopography and underlying structures. A multi-spectral image is one thatcaptures image data within specific wavelength ranges across theelectromagnetic spectrum. The wavelengths may be separated by filters orby the use of instruments that are sensitive to particular wavelengths,including light from frequencies beyond the visible light range, e.g.,IR and ultraviolet. Spectral imaging can allow extraction of additionalinformation the human eye fails to capture with its receptors for red,green, and blue. The use of multi-spectral imaging is described ingreater detail under the heading “Advanced Imaging Acquisition Module”in .S. Patent Application Publication No. US 2019-0200844 A1 (U.S.patent application Ser. No. 16/209,385), titled METHOD OF HUBCOMMUNICATION, PROCESSING, STORAGE AND DISPLAY, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.Multi-spectrum monitoring can be a useful tool in relocating a surgicalfield after a surgical task is completed to perform one or more of thepreviously described tests on the treated tissue. It is axiomatic thatstrict sterilization of the operating room and surgical equipment isrequired during any surgery. The strict hygiene and sterilizationconditions required in a “surgical theater,” i.e., an operating ortreatment room, necessitate the highest possible sterility of allmedical devices and equipment. Part of that sterilization process is theneed to sterilize anything that comes in contact with the patient orpenetrates the sterile field, including the imaging device 124 and itsattachments and components. It will be appreciated that the sterilefield may be considered a specified area, such as within a tray or on asterile towel, that is considered free of microorganisms, or the sterilefield may be considered an area, immediately around a patient, who hasbeen prepared for a surgical procedure. The sterile field may includethe scrubbed team members, who are properly attired, and all furnitureand fixtures in the area.

Referring now to FIG. 3, a hub 106 is depicted in communication with avisualization system 108, a robotic system 110, and a handheldintelligent surgical instrument 112. The hub 106 includes a hub display135, an imaging module 138, a generator module 140, a communicationmodule 130, a processor module 132, a storage array 134, and anoperating-room mapping module 133. In certain aspects, as illustrated inFIG. 3, the hub 106 further includes a smoke evacuation module 126and/or a suction/irrigation module 128. During a surgical procedure,energy application to tissue, for sealing and/or cutting, is generallyassociated with smoke evacuation, suction of excess fluid, and/orirrigation of the tissue. Fluid, power, and/or data lines from differentsources are often entangled during the surgical procedure. Valuable timecan be lost addressing this issue during a surgical procedure.Detangling the lines may necessitate disconnecting the lines from theirrespective modules, which may require resetting the modules. The hubmodular enclosure 136 offers a unified environment for managing thepower, data, and fluid lines, which reduces the frequency ofentanglement between such lines. Aspects of the present disclosurepresent a surgical hub for use in a surgical procedure that involvesenergy application to tissue at a surgical site. The surgical hubincludes a hub enclosure and a combo generator module slidablyreceivable in a docking station of the hub enclosure. The dockingstation includes data and power contacts. The combo generator moduleincludes two or more of an ultrasonic energy generator component, abipolar RF energy generator component, and a monopolar RF energygenerator component that are housed in a single unit. In one aspect, thecombo generator module also includes a smoke evacuation component, atleast one energy delivery cable for connecting the combo generatormodule to a surgical instrument, at least one smoke evacuation componentconfigured to evacuate smoke, fluid, and/or particulates generated bythe application of therapeutic energy to the tissue, and a fluid lineextending from the remote surgical site to the smoke evacuationcomponent. In one aspect, the fluid line is a first fluid line and asecond fluid line extends from the remote surgical site to a suction andirrigation module slidably received in the hub enclosure. In one aspect,the hub enclosure comprises a fluid interface. Certain surgicalprocedures may require the application of more than one energy type tothe tissue. One energy type may be more beneficial for cutting thetissue, while another different energy type may be more beneficial forsealing the tissue. For example, a bipolar generator can be used to sealthe tissue while an ultrasonic generator can be used to cut the sealedtissue. Aspects of the present disclosure present a solution where a hubmodular enclosure 136 is configured to accommodate different generators,and facilitate an interactive communication therebetween. One of theadvantages of the hub modular enclosure 136 is enabling the quickremoval and/or replacement of various modules. Aspects of the presentdisclosure present a modular surgical enclosure for use in a surgicalprocedure that involves energy application to tissue. The modularsurgical enclosure includes a first energy-generator module, configuredto generate a first energy for application to the tissue, and a firstdocking station comprising a first docking port that includes first dataand power contacts, wherein the first energy-generator module isslidably movable into an electrical engagement with the power and datacontacts and wherein the first energy-generator module is slidablymovable out of the electrical engagement with the first power and datacontacts. Further to the above, the modular surgical enclosure alsoincludes a second energy-generator module con-figured to generate asecond energy, different than the first energy, for application to thetissue, and a second docking station comprising a second docking portthat includes second data and power contacts, wherein the secondenergy-generator module is slidably movable into an electricalengagement with the power and data contacts, and wherein the secondenergy-generator module is slidably movable out of the electricalengagement with the second power and data contacts. In addition, themodular surgical enclosure also includes a communication bus between thefirst docking port and the second docking port, configured to facilitatecom-munication between the first energy-generator module and the secondenergy-generator module. Referring to FIG. 3, aspects of the presentdisclosure are presented for a hub modular enclosure 136 that allows themodular integration of a generator module 140, a smoke evacuation module126, and a suction/irrigation module 128. The hub modular enclosure 136further facilitates interactive communication between the modules 140,126, 128. The generator module 140 can be a generator module withintegrated monopolar, bipolar, and ultrasonic components supported in asingle housing unit slidably insertable into the hub modular enclosure136. The generator module 140 can be configured to connect to amonopolar device 142, a bipolar device 144, and an ultrasonic device146. Alternatively, the generator module 140 may comprise a series ofmonopolar, bipolar, and/or ultrasonic generator modules that interactthrough the hub modular enclosure 136. The hub modular enclosure 136 canbe configured to facilitate the insertion of multiple generators andinteractive communication between the generators docked into the hubmodular enclosure 136 so that the generators would act as a singlegenerator.

FIG. 4 illustrates a surgical data network 201 comprising a modularcommunication hub 203 configured to connect modular devices located inone or more operating theaters of a healthcare facility, or any room ina healthcare facility specially equipped for surgical operations, to acloud-based system (e.g., the cloud 204 that may include a remote server213 coupled to a storage device 205). In one aspect, the modularcommunication hub 203 comprises a network hub 207 and/or a networkswitch 209 in communication with a network router. The modularcommunication hub 203 also can be coupled to a local computer system 210to provide local computer processing and data manipulation. The surgicaldata network 201 may be configured as passive, intelligent, orswitching. A passive surgical data network serves as a conduit for thedata, enabling it to go from one device (or segment) to another and tothe cloud computing resources. An intelligent surgical data networkincludes additional features to enable the traffic passing through thesurgical data network to be monitored and to configure each port in thenetwork hub 207 or network switch 209. An intelligent surgical datanetwork may be referred to as a manageable hub or switch. A switchinghub reads the destination address of each packet and then forwards thepacket to the correct port.

Modular devices 1 a-1 n located in the operating theater may be coupledto the modular communication hub 203. The network hub 207 and/or thenetwork switch 209 may be coupled to a network router 211 to connect thedevices 1 a-1 n to the cloud 204 or the local computer system 210. Dataassociated with the devices 1 a-1 n may be transferred to cloud-basedcomputers via the router for remote data processing and manipulation.Data associated with the devices 1 a-1 n may also be transferred to thelocal computer system 210 for local data processing and manipulation.Modular devices 2 a-2 m located in the same operating theater also maybe coupled to a network switch 209. The network switch 209 may becoupled to the network hub 207 and/or the network router 211 to connectto the devices 2 a-2 m to the cloud 204. Data associated with thedevices 2 a-2 n may be transferred to the cloud 204 via the networkrouter 211 for data processing and manipulation. Data associated withthe devices 2 a-2 m may also be transferred to the local computer system210 for local data processing and manipulation.

It will be appreciated that the surgical data network 201 may beexpanded by interconnecting multiple network hubs 207 and/or multiplenetwork switches 209 with multiple network routers 211. The modularcommunication hub 203 may be contained in a modular control towerconfigured to receive multiple devices 1 a-1 n/2 a-2 m. The localcomputer system 210 also may be contained in a modular control tower.The modular communication hub 203 is connected to a display 212 todisplay images obtained by some of the devices 1 a-1 n/2 a-2 m, forexample during surgical procedures. In various aspects, the devices 1a-1 n/2 a-2 m may include, for example, various modules such as animaging module 138 coupled to an endoscope, a generator module 140coupled to an energy-based surgical device, a smoke evacuation module126, a suction/irrigation module 128, a communication module 130, aprocessor module 132, a storage array 134, a surgical device coupled toa display, and/or a non-contact sensor module, among other modulardevices that may be connected to the modular communication hub 203 ofthe surgical data network 201.

In one aspect, the surgical data network 201 may comprise a combinationof network hub(s), network switch(es), and network router(s) connectingthe devices 1 a-1 n/2 a-2 m to the cloud. Any one of or all of thedevices 1 a-1 n/2 a-2 m coupled to the network hub or network switch maycollect data in real time and transfer the data to cloud computers fordata processing and manipulation. It will be appreciated that cloudcomputing relies on sharing computing resources rather than having localservers or personal devices to handle software applications. The word“cloud” may be used as a metaphor for “the Internet,” although the termis not limited as such. Accordingly, the term “cloud computing” may beused herein to refer to “a type of Internet-based computing,” wheredifferent services—such as servers, storage, and applications—aredelivered to the modular communication hub 203 and/or computer system210 located in the surgical theater (e.g., a fixed, mobile, temporary,or field operating room or space) and to devices connected to themodular communication hub 203 and/or computer system 210 through theInternet. The cloud infrastructure may be maintained by a cloud serviceprovider. In this context, the cloud service provider may be the entitythat coordinates the usage and control of the devices 1 a-1 n/2 a-2 mlocated in one or more operating theaters. The cloud computing servicescan perform a large number of calculations based on the data gathered bysmart surgical instruments, robots, and other computerized deviceslocated in the operating theater. The hub hardware enables multipledevices or connections to be connected to a computer that communicateswith the cloud computing resources and storage.

Applying cloud computer data processing techniques on the data collectedby the devices 1 a-1 n/2 a-2 m, the surgical data network can provideimproved surgical outcomes, reduced costs, and improved patientsatisfaction. At least some of the devices 1 a-1 n/2 a-2 m may beemployed to view tissue states to assess leaks or perfusion of sealedtissue after a tissue sealing and cutting procedure. At least some ofthe devices 1 a-1 n/2 a-2 m may be employed to identify pathology, suchas the effects of diseases, using the cloud-based computing to examinedata including images of samples of body tissue for diagnostic purposes.This may include localization and margin confirmation of tissue andphenotypes. At least some of the devices 1 a-1 n/2 a-2 m may be employedto identify anatomical structures of the body using a variety of sensorsintegrated with imaging devices and techniques such as overlaying imagescaptured by multiple imaging devices. The data gathered by the devices 1a-1 n/2 a-2 m, including image data, may be transferred to the cloud 204or the local computer system 210 or both for data processing andmanipulation including image processing and manipulation. The data maybe analyzed to improve surgical procedure outcomes by determining iffurther treatment, such as the application of endoscopic intervention,emerging technologies, a targeted radiation, targeted intervention, andprecise robotics to tissue-specific sites and conditions, may bepursued. Such data analysis may further employ outcome analyticsprocessing, and using standardized approaches may provide beneficialfeedback to either confirm surgical treatments and the behavior of thesurgeon or suggest modifications to surgical treatments and the behaviorof the surgeon.

The operating theater devices 1 a-1 n may be connected to the modularcommunication hub 203 over a wired channel or a wireless channeldepending on the configuration of the devices 1 a-1 n to a network hub.The network hub 207 may be implemented, in one aspect, as a localnetwork broadcast device that works on the physical layer of the OpenSystem Interconnection (OSI) model. The network hub may provideconnectivity to the devices 1 a-1 n located in the same operatingtheater network. The network hub 207 may collect data in the form ofpackets and sends them to the router in half duplex mode. The networkhub 207 may not store any media access control/Internet Protocol(MAC/IP) to transfer the device data. Only one of the devices 1 a-1 ncan send data at a time through the network hub 207. The network hub 207may not have routing tables or intelligence regarding where to sendinformation and broadcasts all network data across each connection andto a remote server 213 (FIG. 4) over the cloud 204. The network hub 207can detect basic network errors such as collisions, but having allinformation broadcast to multiple ports can be a security risk and causebottlenecks.

The operating theater devices 2 a-2 m may be connected to a networkswitch 209 over a wired channel or a wireless channel. The networkswitch 209 works in the data link layer of the OSI model. The networkswitch 209 may be a multicast device for connecting the devices 2 a-2 mlocated in the same operating theater to the network. The network switch209 may send data in the form of frames to the network router 211 andworks in full duplex mode. Multiple devices 2 a-2 m can send data at thesame time through the network switch 209. The network switch 209 storesand uses MAC addresses of the devices 2 a-2 m to transfer data.

The network hub 207 and/or the network switch 209 may be coupled to thenetwork router 211 for connection to the cloud 204. The network router211 works in the network layer of the OSI model. The network router 211creates a route for transmitting data packets received from the networkhub 207 and/or network switch 211 to cloud-based computer resources forfurther processing and manipulation of the data collected by any one ofor all the devices 1 a-1 n/2 a-2 m. The network router 211 may beemployed to connect two or more different networks located in differentlocations, such as, for example, different operating theaters of thesame healthcare facility or different networks located in differentoperating theaters of different healthcare facilities. The networkrouter 211 may send data in the form of packets to the cloud 204 andworks in full duplex mode. Multiple devices can send data at the sametime. The network router 211 uses IP addresses to transfer data.

In an example, the network hub 207 may be implemented as a USB hub,which allows multiple USB devices to be connected to a host computer.The USB hub may expand a single USB port into several tiers so thatthere are more ports available to connect devices to the host systemcomputer. The network hub 207 may include wired or wireless capabilitiesto receive information over a wired channel or a wireless channel. Inone aspect, a wireless USB short-range, high-bandwidth wireless radiocommunication protocol may be employed for communication between thedevices 1 a-1 n and devices 2 a-2 m located in the operating theater.

In examples, the operating theater devices 1 a-1 n/2 a-2 m maycommunicate to the modular communication hub 203 via Bluetooth wirelesstechnology standard for exchanging data over short distances (usingshort-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz)from fixed and mobile devices and building personal area networks(PANs). The operating theater devices 1 a-1 n/2 a-2 m may communicate tothe modular communication hub 203 via a number of wireless or wiredcommunication standards or protocols, including but not limited to Wi-Fi(IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio(NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE,GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, as wellas any other wireless and wired protocols that are designated as 3G, 4G,5G, and beyond. The computing module may include a plurality ofcommunication modules. For instance, a first communication module may bededicated to shorter-range wireless communications such as Wi-Fi andBluetooth, and a second communication module may be dedicated tolonger-range wireless communications such as GPS, EDGE, GPRS, CDMA,WiMAX, LTE, Ev-DO, and others.

The modular communication hub 203 may serve as a central connection forone or all of the operating theater devices 1 a-1 n/2 a-2 m and mayhandle a data type known as frames. Frames may carry the data generatedby the devices 1 a-1 n/2 a-2 m. When a frame is received by the modularcommunication hub 203, it is amplified and transmitted to the networkrouter 211, which transfers the data to the cloud computing resources byusing a number of wireless or wired communication standards orprotocols, as described herein.

The modular communication hub 203 can be used as a standalone device orbe connected to compatible network hubs and network switches to form alarger network. The modular communication hub 203 can be generally easyto install, configure, and maintain, making it a good option fornetworking the operating theater devices 1 a-1 n/2 a-2 m.

FIG. 5 illustrates a computer-implemented interactive surgical system200. The computer-implemented interactive surgical system 200 is similarin many respects to the computer-implemented interactive surgical system100. For example, the computer-implemented interactive surgical system200 includes one or more surgical systems 202, which are similar in manyrespects to the surgical systems 102. Each surgical system 202 includesat least one surgical hub 206 in communication with a cloud 204 that mayinclude a remote server 213. In one aspect, the computer-implementedinteractive surgical system 200 comprises a modular control tower 236connected to multiple operating theater devices such as, for example,intelligent surgical instruments, robots, and other computerized deviceslocated in the operating theater. As shown in FIG. 6, the modularcontrol tower 236 comprises a modular communication hub 203 coupled to acomputer system 210.

As illustrated in the example of FIG. 5, the modular control tower 236may be coupled to an imaging module 238 that may be coupled to anendoscope 239, a generator module 240 that may be coupled to an energydevice 241, a smoke evacuator module 226, a suction/irrigation module228, a communication module 230, a processor module 232, a storage array234, a smart device/instrument 235 optionally coupled to a display 237,and a non-contact sensor module 242. The operating theater devices maybe coupled to cloud computing resources and data storage via the modularcontrol tower 236. A robot hub 222 also may be connected to the modularcontrol tower 236 and to the cloud computing resources. Thedevices/instruments 235, visualization systems 208, among others, may becoupled to the modular control tower 236 via wired or wirelesscommunication standards or protocols, as described herein. The modularcontrol tower 236 may be coupled to a hub display 215 (e.g., monitor,screen) to display and overlay images received from the imaging module,device/instrument display, and/or other visualization systems 208. Thehub display also may display data received from devices connected to themodular control tower in conjunction with images and overlaid images.

FIG. 6 illustrates a surgical hub 206 comprising a plurality of modulescoupled to the modular control tower 236. The modular control tower 236may comprise a modular communication hub 203, e.g., a networkconnectivity device, and a computer system 210 to provide localprocessing, visualization, and imaging, for example. As shown in FIG. 6,the modular communication hub 203 may be connected in a tieredconfiguration to expand the number of modules (e.g., devices) that maybe connected to the modular communication hub 203 and transfer dataassociated with the modules to the computer system 210, cloud computingresources, or both. As shown in FIG. 6, each of the networkhubs/switches in the modular communication hub 203 may include threedownstream ports and one upstream port. The upstream network hub/switchmay be connected to a processor to provide a communication connection tothe cloud computing resources and a local display 217. Communication tothe cloud 204 may be made either through a wired or a wirelesscommunication channel.

The surgical hub 206 may employ a non-contact sensor module 242 tomeasure the dimensions of the operating theater and generate a map ofthe surgical theater using either ultrasonic or laser-type non-contactmeasurement devices. An ultrasound-based non-contact sensor module mayscan the operating theater by transmitting a burst of ultrasound andreceiving the echo when it bounces off the perimeter walls of anoperating theater as described under the heading “Surgical Hub SpatialAwareness Within an Operating Room” in S. Patent Application PublicationNo. US 2019-0200844 A1 (U.S. patent application Ser. No. 16/209,385),titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,filed Dec. 4, 2018, which is herein incorporated by reference in itsentirety, in which the sensor module is configured to determine the sizeof the operating theater and to adjust Bluetooth-pairing distancelimits. A laser-based non-contact sensor module may scan the operatingtheater by transmitting laser light pulses, receiving laser light pulsesthat bounce off the perimeter walls of the operating theater, andcomparing the phase of the transmitted pulse to the received pulse todetermine the size of the operating theater and to adjust Bluetoothpairing distance limits, for example.

The computer system 210 may comprise a processor 244 and a networkinterface 245. The processor 244 can be coupled to a communicationmodule 247, storage 248, memory 249, non-volatile memory 250, andinput/output interface 251 via a system bus. The system bus can be anyof several types of bus structure(s) including the memory bus or memorycontroller, a peripheral bus or external bus, and/or a local bus usingany variety of available bus architectures including, but not limitedto, 9-bit bus, Industrial Standard Architecture (ISA), Micro-CharmelArchitecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics(IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI),USB, Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), Small Computer Systems Interface(SCSI), or any other proprietary bus.

The processor 244 may be any single-core or multicore processor such asthose known under the trade name ARM Cortex by Texas Instruments. In oneaspect, the processor may be an LM4F230H5QR ARM Cortex-M4F ProcessorCore, available from Texas Instruments, for example, comprising anon-chip memory of 256 KB single-cycle flash memory, or othernon-volatile memory, up to 40 MHz, a prefetch buffer to improveperformance above 40 MHz, a 32 KB single-cycle serial random accessmemory (SRAM), an internal read-only memory (ROM) loaded withStellarisWare® software, a 2 KB electrically erasable programmableread-only memory (EEPROM), and/or one or more pulse width modulation(PWM) modules, one or more quadrature encoder inputs (QEI) analogs, oneor more 12-bit analog-to-digital converters (ADCs) with 12 analog inputchannels, details of which are available for the product datasheet.

In one aspect, the processor 244 may comprise a safety controllercomprising two controller-based families such as TMS570 and RM4x, knownunder the trade name Hercules ARM Cortex R4, also by Texas Instruments.The safety controller may be configured specifically for IEC 61508 andISO 26262 safety critical applications, among others, to provideadvanced integrated safety features while delivering scalableperformance, connectivity, and memory options.

The system memory may include volatile memory and non-volatile memory.The basic input/output system (BIOS), containing the basic routines totransfer information between elements within the computer system, suchas during start-up, is stored in non-volatile memory. For example, thenon-volatile memory can include ROM, programmable ROM (PROM),electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatilememory includes random-access memory (RAM), which acts as external cachememory. Moreover, RAM is available in many forms such as SRAM, dynamicRAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and directRambus RAM (DRRAM).

The computer system 210 also may include removable/non-removable,volatile/non-volatile computer storage media, such as for example diskstorage. The disk storage can include, but is not limited to, deviceslike a magnetic disk drive, floppy disk drive, tape drive, Jaz drive,Zip drive, LS-60 drive, flash memory card, or memory stick. In addition,the disk storage can include storage media separately or in combinationwith other storage media including, but not limited to, an optical discdrive such as a compact disc ROM device (CD-ROM), compact discrecordable drive (CD-R Drive), compact disc rewritable drive (CD-RWDrive), or a digital versatile disc ROM drive (DVD-ROM). To facilitatethe connection of the disk storage devices to the system bus, aremovable or non-removable interface may be employed.

It is to be appreciated that the computer system 210 may includesoftware that acts as an intermediary between users and the basiccomputer resources described in a suitable operating environment. Suchsoftware may include an operating system. The operating system, whichcan be stored on the disk storage, may act to control and allocateresources of the computer system. System applications may take advantageof the management of resources by the operating system through programmodules and program data stored either in the system memory or on thedisk storage. It is to be appreciated that various components describedherein can be implemented with various operating systems or combinationsof operating systems.

A user may enter commands or information into the computer system 210through input device(s) coupled to the I/O interface 251. The inputdevices may include, but are not limited to, a pointing device such as amouse, trackball, stylus, touch pad, keyboard, microphone, joystick,game pad, satellite dish, scanner, TV tuner card, digital camera,digital video camera, web camera, and the like. These and other inputdevices connect to the processor through the system bus via interfaceport(s). The interface port(s) include, for example, a serial port, aparallel port, a game port, and a USB. The output device(s) use some ofthe same types of ports as input device(s). Thus, for example, a USBport may be used to provide input to the computer system and to outputinformation from the computer system to an output device. An outputadapter may be provided to illustrate that there can be some outputdevices like monitors, displays, speakers, and printers, among otheroutput devices that may require special adapters. The output adaptersmay include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device andthe system bus. It should be noted that other devices and/or systems ofdevices, such as remote computer(s), may provide both input and outputcapabilities.

The computer system 210 can operate in a networked environment usinglogical connections to one or more remote computers, such as cloudcomputer(s), or local computers. The remote cloud computer(s) can be apersonal computer, server, router, network PC, workstation,microprocessor-based appliance, peer device, or other common networknode, and the like, and typically includes many or all of the elementsdescribed relative to the computer system. For purposes of brevity, onlya memory storage device is illustrated with the remote computer(s). Theremote computer(s) may be logically connected to the computer systemthrough a network interface and then physically connected via acommunication connection. The network interface may encompasscommunication networks such as local area networks (LANs) and wide areanetworks (WANs). LAN technologies may include Fiber Distributed DataInterface (FDDI), Copper Distributed Data Interface (CDDI),Ethernet/IEEE 802.3, Token Ring/IEEE 802.5 and the like. WANtechnologies may include, but are not limited to, point-to-point links,circuit-switching networks like Integrated Services Digital Networks(ISDN) and variations thereon, packet-switching networks, and DigitalSubscriber Lines (DSL).

In various aspects, the computer system 210 of FIG. 6, the imagingmodule 238 and/or visualization system 208, and/or the processor module232 of FIGS. 5-6, may comprise an image processor, image-processingengine, media processor, or any specialized digital signal processor(DSP) used for the processing of digital images. The image processor mayemploy parallel computing with single instruction, multiple data (SIMD)or multiple instruction, multiple data (MIMD) technologies to increasespeed and efficiency. The digital image-processing engine can perform arange of tasks. The image processor may be a system on a chip withmulticore processor architecture.

The communication connection(s) may refer to the hardware/softwareemployed to connect the network interface to the bus. While thecommunication connection is shown for illustrative clarity inside thecomputer system, it can also be external to the computer system 210. Thehardware/software necessary for connection to the network interface mayinclude, for illustrative purposes only, internal and externaltechnologies such as modems, including regular telephone-grade modems,cable modems, and DSL modems, ISDN adapters, and Ethernet cards.

FIG. 7. is a block diagram of an example surgical simulator system. Thesurgical simulator system may include a simulation device 30000. Thesurgical simulator system may include an application creation device30002, a human interface device 30004, a surgeon agent device 30006,and/or a surgical data system 30008.

The simulation device 30000 may provide core simulation functionality.For example, the loading/running of one or more simulations, thereception and processing of user control information input, thegeneration and transmission of visual, audible, and/or hapticinformation output, the collection of simulation operation and activityinformation, and the primary simulation cycle processing may beperformed by the simulation device 30000.

The application creation device 30002 may provide simulation authoringfunctionality. Individual simulation applications may be stored asapplication modules 30010 at the simulation device 30000. Theapplication modules 30010 may be created, modified, and/or deleted bythe application creation device 30002. The application modules 30010 mayinclude computer readable and/or executable instructions to direct anoperation of the simulation device 30000. For example, the applicationmodules 30010 may include any filetype suitable for storing informationto run a surgical simulation, for example, simulation scripts,programming code, structure data files such as Extensible MarkupLanguage (XML) files, database files, and the like.

The application creation device 30002 may include a graphical userinterface with controls to author application modules 30010. Theapplication creation device 3002 may communicate with the simulationdevice 30000 to retrieve, modify, and/or load application modules 30010for simulation operation. For example, the graphical user interface mayinclude interface structures to allow a user to select simulationactivities, to input various simulation parameters, to set simulationobjectives, and to confirm simulation execution. The applicationcreation device 30002 may be provided as a stand-alone device and/orintegrated with one or more other devices of the surgical simulationsystem, such as integrated with the simulation device 30000 for example.

The human interface device 30004 may include any hardware, software,and/or combination thereof that enables a human user to interact with asimulation provided by the simulation device 30000. The human interfacedevice 30004 may enable a user to provide control input to thesimulation device 300000 and/or to receive output information (such asvisual, audible, and/or haptic information) from the simulation device30000. In one example, the human interface device 30004 may include atraditional desktop computer.

The human interface device 30004 may include suitable physicalequipment. For example, the human interface device 30004 may includephysical equipment that mimic physically and/or virtually aspects of asurgical procedure. For example, such equipment may include bench-topunits, part-task virtual reality units, high fidelity virtual realityunits, high fidelity full-size patient units, suite units, high fidelityfull operating room units, full physics virtual reality units, surgicalrobot console units, and the like. For example, the human interfacedevice 30004 may include devices such as the computer-based simulatorinterfaces disclosed by Gallager et al, “Simulations for ProceduralTraining,” Fundamentals of Surgical Simulation, Principles and Practice,Springer (2012).

The human interface device 30004 may include physical equipment thatmimics, physically and/or virtually, surgical instruments. For example,the human interface device 30004 may include physical devices that mimicsurgical instruments, appliances, and consumables, such as accessequipment, such as trocars, hand-access ports, insufflation needles, andguiding sheaths; adjunctive hemostats, such as patches, gelatins, andpowders; craniomaxillofacial appliances, like distractors and plates;balloons and inflators; catheters, like diagnostic catheters, accesscatheters, vascular catheters, and therapeutic catheters; energy sealingand dissecting devices, like tissue sealers, shears, blades, andforceps; orthopedic equipment, like reduction wires, compression screws,plates, implants, drills, burrs, rods, and connectors; ligationinstruments, like open and endoscopic clip appliers; microwave ablationequipment; ancillary endoscopic instruments, like drains, sutures,ligature, needle holders, retrievers, and suture clips; surgicalstapling equipment, like open staplers, endoscopic staplers, cutterstaplers, powered staplers, circular staplers, vascular staplers, linearstaplers, staple cartridges, and staple line reinforcement applicators;wound closure materials, like suture, adhesives, needles, and knotlesstissue control devices; imaging devices, like minimally invasive imagingdevices; and the like. For example, the human interface device 30004 mayinclude virtual reality handheld controllers, that when operated with avirtual reality headset, mimics the surgical instruments, appliances,and consumables, such as those disclosed above.

The human interface device 30004 may include a display that communicatesvisual representations of the simulation to the user. The humaninterface device 30004 may include a computer display. The humaninterface device 30004 may include a virtual reality headset display.For example, the virtual reality headset display may be used display thesurgical environment, such as that disclosed in FIG. 2, herein. A userwith such a virtual reality headset display may view and/or interactwith any of the elements in the surgical operating room 116, including,for example, the patient, the robotic system 110, the surgeon's console118, the surgical robotic hub 122, one or more surgical tools 117, theimaging device 124, the patient side cart 120, one or more displays 119,107, 109, and the like.

The human interface device 30006 may present visual information thatrepresents the point of the view of the surgeon. The human interfacedevice 30006 may present visual information from a simulated imagingdevice, such as an arthroscope, angioscope, bronchoscope,choledochoscope, colonoscope, cytoscope, duodenoscope, enteroscope,esophagogastro-duodenoscope (gastroscope), endoscope, laryngoscope,nasopharyngo-neproscope, sigmoidoscope, thoracoscope, ureteroscope, andtheir related instruments, controls, and the like. The human interfacedevice 30006 may present visual information from a simulatedsupplemental intra-operative imaging equipment, like computed tomography(CT) units, magnetic resonance imaging (MRI) units, image-guided surgeryunits, intra-operative ultrasound units; fluoroscopy units, and thelike. Such point-of-view visual information, surgical imaginginformation, and supplemental intra-operative imaging information may bedisplayed in any combination to the user suitable for the simulation'soperation. For example, such information may be presented to the user asa single full-screen view, a tiled window view, a picture-in-a-pictureview, or registered to a simulated display unit in a virtual realityview.

The human interface device 30004 may include a physical and/or virtualreality surgical robot surgeon console. For example, an examplesurgeon-console-like human interface device 30004 may include a display,such as a stereo vision display and control inputs, including hand-heldmanipulators, foot pedals, and the like. For example, thesurgeon-console-like human interface device 30004 may include aninterface of the surgeon's console 118, disclosed herein. The humaninterface device 30004 may enable voice controls via, for example, amicrophone and speech recognition functionality. The human interfacedevice 30004 may provide audible feedback via, for example, a speaker.The human interface device 30004 may provide haptic feedback via, forexample, vibration, force feedback, air vortex rings, and ultrasoundtechniques.

As implemented, the human interface device 30004 may be provided as astand-alone device and/or integrated with one or more other devices ofthe surgical simulation system, such as integrated with the simulationdevice 30000 for example. The simulation device 30000 may include aninterface module 30012 to communicate with the human interface device30004. In an example, human interface device 30004 may be integratedinto one or more elements of the computer-implemented interactivesurgical system 100. For example, the human interface device 30004 maybe integrated into the computer system 210. For example, the humaninterface device 30004 may be integrated into the hub 106. For example,the human interface device 30004 may be integrated into thevisualization system 108. The interface module 30012 may communicatewith the one or more elements of the computer-implemented interactivesurgical system 100 via the surgical data system interface module 30014for example.

In an embodiment, more than one human interface device 30004 mayconcurrently engage with the simulation device 30000. For example, amulti-person simulation application

The surgeon agent device 30006 may include any hardware and/or softwaresuitable for providing a computer-based control and response to theinput and output of the simulation device 30000. The surgeon agentdevice 30006 may include a computer process that mimics human input tothe simulation device 30000. For example, the surgeon agent device 30006may be able to record and register control inputs, such as basicinstrument manipulation. The surgeon agent device 30006 may include acomputer process that can access a input/output application programminginterface (API) of the simulation device 30000. For example, the API mayreveal one or more input/output functions that may be directed accordingto the surgeon agent device 3006. The functions may include granularmanipulation and physics-based input/output functions, such as functionsthat directly control the location and movement of instruments. Thefunctions may include less granular surgical-activity-based input/outputfunctions, such as a ligation activity, a suturing activity, a staplingactivity, and the like. The functions may include less granular surgicaltask and/or stage-based input/output functions, such as surgical accessfunction, organ mobilization function, and the like. Each function mayinclude parameter consistent with its level of granularity. Theparameters may provide specific details to direct the operation of thefunction within the simulation. The surgeon agent 30006 may includefunctionality for generating and operating multiple simulation runs. Forexample, a user may wish to estimate the duration of various suturingtechniques. A surgeon agent device 30006 may be used to script thesimulation of any number of different techniques, each of which can berun via the simulation device, and the metrics collected by thesimulation device may be used to estimate the difference in durations.

The surgeon agent device 30006 may be provided as a stand along deviceand/or integrated with one or more other devices of the surgicalsimulation system, such as integrated with the simulation device 30000for example. The simulation device 30000 may include an interface module30012 to communicate with the surgeon agent device 30006. For example,the surgeon agent device 30006 may be integrated as a module of thesimulation device 30000. For example, the surgeon agent device 30006 maybe integrated into an application module 30010 of the simulation device.

The surgical data system 30008 may include any hardware and/or softwaresuitable for providing external, structured surgical information andfunctionality to the simulation device 30000. The surgical data system30008 may include the structure and/or functions described in connectionwith FIGS. 1-6 herein. For example, the surgical data system 30008 mayinclude one or more elements of a computer-implemented interactivesurgical system 100. The surgical data system 30008 may include, forexample, a surgical hub 106. For example, the simulation device 30000include a surgical data system interface module 30014 that enablescommunication with the surgical hub 106 via the surgical hub'scommunication module 130. The surgical data system 30008 may include,for example, on or more surgical data repositories. For example, thesurgical data system 30008 may include the computer system 210 locatedin the surgical theater. For example, the surgical data system 30008 mayinclude the remote server 213 in the cloud 204.

A surgical data system 30008, such as the surgical hub 106 for example,may provide data to the simulation device 30000 and/or the applicationcreation device 30002. For example, the data may include any surgicaldata collected and/or generated by the surgical hub 106. Also forexample, the simulation device 30000 may receive similar data directlyfrom any of the networked devices disclosed in FIGS. 1-6. Such data mayinclude information about a live surgical procedure, for example. Suchdata may include information about a past surgical procedure. Such datamay include information about future, scheduled surgical procedures.

Information about the surgical procedures may include information aboutthe patient, the staff, the procedure as planned, the procedure asexperienced, and post-operative activity including patient outcomes. Forexample, the information received and used by the simulation device mayinclude patient records, patient imaging, models of patient anatomy,patient lab results, patient medical history, and the like. For example,the information received and used by the simulation device may include astaff manifest for a procedure, details about the past procedures of thespecific staff members, staff metrics, experience, recent scheduling andworkload, and historical surgical activity, such instrument usestatistics, procedure duration, and the like. For example, theinformation received and used by the simulation device may includeprocedure plans, equipment and inventory information, pull-lists,checklists, procedure plan analysis and recommendations. For example,the information received and used by the simulation device may includeany data collected or generated during a live procedure, such asprocedure progress, milestones, patient information, vitals, operatingtheater setup, staff movement, imaging, instrument use, surgicaltechnique, such as that captured by video, recorded manually, and/orinferred from smart-instrument reporting for example, duration, abnormalevent reporting, and the like. Any data captured during a live proceduremay also be stored and made available as a past procedure. For example,the information received and used by the simulation device may includepost-operative records, patient recovery information, and patientoutcome information, post-operative diagnostic information, such aslabs, imaging, etc.,

The simulation device 30000 may include any computer or processingplatform suitable for executing one or more simulations. The simulationmay include a computer-modeled environment of a surgical procedure. Forexample, the simulation may include a model of a patient's anatomyand/or physiology. For example, the simulation may include a model ofthe actions and/or instruments of one or more healthcare professionals,such as the actions of a surgeon, nurse, other doctor, technician, orthe like.

The simulation device 30000 may include one or more functional modules.Each module may include hardware, software, or a combination thereofthat enable functionality of the module. One or more modules, operatingin concert, may represent a computer framework on which a simulation ofa medical procedure may be executed. The modules may include hardwareelements, such as a computer processing unit, a graphics processingunit, a field-programmable gate array (FPGAs), communications hardware,memory, and the like. The modules may include software elements thatwhen executed by a processor cause the module to perform certainfunctions.

The simulation device may include a core simulation module 30016, asimulation applications module directory 30018, the interface module30012, an object properties module 30020, a physics module 30022, aphysiology model 30024, a texture model 30026, a 3D graphics pipeline30028, the surgical data system interface module 30014, a metricsextraction module 30030, a session storage and management module 30032,for example. The simulation device may include an operating systemmodule 30034.

The core simulation model 30016 may provide primary simulationfunctionality of the simulation device 30000. For example, the coresimulation module 30016 may include code for initializing a simulation,for communicating and interacting with other modules of the simulationdevice 30000, and/or for managing architectural level simulationparameters. For example, the core simulation module 30016 may include amaster event clock to provide time alignment and/or coordination of theoperation of the modules of the simulation device 30000. For example,the core simulation module 30016 may establish the overall simulationframe rate.

The core simulation module 30016 may include core for providing a mastersimulation cycle. The core simulation module 30016 may run one or moreiteration of the master simulation cycle. Each iteration of the mastersimulation cycle may represent an individual time slice for simulation.In an example, the core simulation module 30016 may run the mastersimulation cycle according to the flow disclosed in FIG. 10.

The simulation applications module directory 30018 may manage thestoring, retrieving, and/or linking of the one or more applicationmodules 30010. Each application module 30010 may include code thatdirects the application-level aspects of a simulation. For example, anapplication module 30010 may include the functionality to provide asimulation of specific anatomy, of specific teaching scope, of specificequipment, or the like. In an example simulation device 30000, anapplication-specific simulation device 30000 may operate with a singleapplication module 30010 with or without a simulation application moduledirectory 30010. The simulation application module directory 30018 mayoperate based on interaction with the core simulation module 30016and/or the application creation device 30002.

The interface module 30012 may provide functionality for interactingwith the human interface device 30004 and/or the surgeon agent device30006. For example, the interface module 30012 may include one or moredrivers to translate information received from human interface device30004 into software commands, interrupts, and the like. For example, theinterface module 30012 may include a software application programminginterface (API) for interacting with the surgeon agent 30006. Theinterface module 30012 may provide information received from the humaninterface module 30004 and/or the surgeon agent device 30006 to othermodules of the simulation device 30000. For example, the interfacemodule 30012 may receive a control input from the human interface module30004 and/or the surgeon agent device 30006 that represents movement ofa simulated instrument and provide that information to one or more othermodules of the simulation device 30000 so the movement may berepresented in the simulation.

The interface module 30012 may provide the API to enable a more granularinteraction with the surgeon agent device 30006. For example, the APImay provide an interface to receive simulation parameters and simulationsettings from the surgeon agent device 30006. Such simulation parametersand/or simulation settings may be like those input by the user via theapplication creation device 30002, for example. For example, the surgeonagent device 30006 may be enabled to run one or more computer-controlledsimulation trials through the simulation device 30000. For example, thesurgeon agent device 30006 may be enabled to run multiple simulations,each with alternative interactions.

The interface module 30012 may send output from the simulation device30000 to the human interface device 30004 and/or the surgeon agentdevice 30006. For example, the output may include visual output, hapticoutput, audio output, and/or structured data output, or the like.

The object properties module 30020 may provide functionality formanaging the simulated appearance and/or behavior of objects within inthe simulation. Simulated objects may include objects such as anatomy,instrument, equipment, consumables, fluids, and the like. An object'sappearance may be managed by object properties, such as location,dimensions, scale, material, parent/child relationships, vertices,faces, interactivity, transparency, trajectory, rendering properties,textures, surface reflectivity, motion blur, layering, and the like. Anobject's behavior may be managed by object properties, such as physicsproperties, mass, motion, collision behavior, elasticity, viscosity,surface tension, rigging constraints, hardness, shear strength, tearingbehavior, grain, and the like.

The physics module 30022 may provide functionality to calculate thephysical responses and/or interaction of objects within the simulation.The physical module may determine such responses and/or interactionsaccording to classical mechanics, fluid mechanics, soft body dynamics,Brownian motion, collision detection, cloth behavior, finite elementanalysis, and the like. The physics module 30022 may include commercialand/or open-source modules, such as PhysX™, Simulation Open FrameworkArchitecture (SOFA)™, VisSim™, and the like.

The physiology module 30024 may provide functionality to calculatephysiological responses and/or interactions of the anatomy and/orpatient as a whole in the simulation. The physiology module 30024 mayprovide physiological models for key organs and/or systems. Thephysiological models may include mathematical models, statisticalmodels, or the like. For example, the physiology module 30024 may modulethe patient's vitals to calculate their response and/or interaction toactivities performed during the simulation. For example, a circulatorymodel may calculate blood pressure in response to a severed vessel inthe simulation. The physiology module 30024 and the physics module 30022may coordinate with each other during the calculation of each state ofthe simulation. For example, blood pressure calculated by thecirculatory model may be used to determine fluid dynamics propertiescalculated by the physics module 30022 and managed by the objectproperties module 30020.

The texture module 30026 may provide functionality to determine,retrieve, and/or generate the appropriate surfacing of objects withinthe simulation. The texture module 30026 may include one or moresurfacing modalities that may be controlled according to parameters ofthe simulation. The surfacing modalities may include artificiallygenerated surfaces, surfaces based on real-world imagery, andcombinations thereof. The texture module 30026 may coordinate operationwith the physics module 30022 to provide accurate haptic feedback to theuser via the user interface module 30012.

The 3D graphics pipeline 30028 may provide functionality for visualrendering of the simulation environment. The 3D graphics pipeline 30028may receive object properties and a perspective. The 3D graphicspipeline 30028 may determine the visualization to be presented to theuser that represents the objects in 3D space as viewed from the cameraperspective. The 3D graphics pipeline 30028 may determine geometricaspects of the rendering, such as lighting, projection, clipping, viewtransformation, and the like. The 3D graphics pipeline 30028 maydetermine rasterization aspects of the rendering, such as fragmentation,pixel shading, vertex shading, geometry sharing, texture filtering, andthe like. The 3D graphics pipeline 30028 may coordinate with the texturemodule 30026 to provide accurate visual feedback to the user via theinterface module 30012.

The surgical data system interface module 30014 may provide interactiveconnectivity to one or more elements of computer-implemented interactivesurgical system 100. Information from the one or more elements of thecomputer-implemented interactive surgical system 100 may be communicatedvia the surgical data system interface module 30014 to one more modulesof the simulation device 30000 to influence operation of a simulation.For example, the surgical data system interface module 30014 may receiveinformation about a surgical procedure an communicate it to acorresponding application module 30010. For example, the surgical datasystem interface module 30014 may receive information about aninstrument and communicate it to the object properties module 30020. Forexample, the surgical data system interface module 30014 may receiveinformation about a patient and communicate to the physiology module.For example, the surgical data system interface module 30014 may receiveinformation about tissue imaging and communicate it to the texturemodule 30026.

Information from the modules of the simulation device 30000 may beprovided, via the surgical data system interface 30014, to one or moreelements of the computer-implemented interactive surgical system 100.For example, one or more elements of the computer-implementedinteractive surgical system 100 may receive statistics related to asimulated procedure plan from the metrics extraction module 30030. Forexample, one or more elements of the computer-implemented interactivesurgical system 100 may receive replayed simulation visualizationprocedure plan from the session storage and management module 30032. Forexample, the surgical data system interface module 30014 may provide acommunications pathway between the interface module 30012 and one ormore elements of the computer-implemented interactive surgical system100. For example, a surgeon during a live surgical procedure may accesssimulation information and/or operate a simulation from the operatingtheater. For example, a surgeon may use the surgeon console 118 toaccess and/or interact with a simulation that corresponds to the livesurgical procedure.

The metrics extraction module 30014 may provide recording functionalityof various parameters related to the operation of the simulation. Forexample, the metrics extraction module 30014 may record metrics relatedto the simulation as a whole, such as duration, number of activities,number of movements, complexity of movements, staff employed, staffmovement, equipment and/or instrument changes, etc. For example, themetrics extraction module 30014 may record metrics related to aparticular aspect of the simulation, such as simulated patient vitals,complications, collisions, bleeding, etc. The metrics extraction module30014 may maintain a master log of metric-related events during asimulation. For metrics extraction module 30014 may recordmetric-related events according to a configuration from the applicationmodule 30010 employed for the simulation.

The session storage and management module 30032 may provide managementfunctionality of the main simulation run-record. For example, thesession storage and management module 30032 may store the information toenable a simulation to be rerun, viewed, and/or analyzed in itsentirety. The session storage and management module 30032 may store theinformation about each input, simulation state, and output, such as theinput, simulation state, and output disclosed with regard to FIG. 10.The session storage and management module 30032 may enable a previoussimulation to be recalled, copied, and initialized with new user input.To illustrate, a surgeon in training may recall a simulation run by anexperienced surgeon, pause the simulation at a critical step, andattempt that step on her own. The session storage and management module30032 may provide overlay functionality between various runs of aparticular simulation. Such overlays may highlight similarities anddifferences and may enhance training.

The operating system module 30034 may manage the hardware and/orsoftware resources for the simulation device 30000. The operating systemmodule 30034 may provide common computing system-level services for theother modules of simulation device 30000. For example, the operatingsystem module 30034 may provide hardware input and output handling,memory allocation, hardware interrupt handling, software interrupthandling, thread processing, single task handling, multi-task handling,and the like. The simulation device 30000 may be a real-time computingdevice. The operating system module 30034 may include a real-timeoperating system. For example, the operating system module 30034 may bedriven by the events and frame rate established by the core simulationmodule 30016.

FIG. 8 is a block diagram of an example surgical simulator system. Thesimulation device 30000 is depicted with an example hardwarearchitecture. For example, the simulation device 30000 may include aprocessor 30034, a memory 30036, a storage 30038, a display adapter30040, a manipulation interface adapter 30042, a surgical data systemadapter 30044, and/or a network adapter 30046. One or more of theprocessor 30034, a memory 30036, a storage 30038, a display adapter30040, a manipulation interface adapter 30042, a surgical data systemadapter 30044, and/or a network adapter 30046 may be used to enableoperation of the modules of the simulation device 30000 disclosedherein.

The processor 30046 may include computer processing unit, graphicsprocessing unit, any suitable microcontroller, microprocessor, fieldprogrammable gate array (FPGA), application specific integrated circuit(ASIC), or the like, and/or any combination thereof that is suitable forprocessing and delivering a 3D simulated environment for interactionwith a computer agent and/or human user. In one example, the processor30046 may include one or more processing units. The processor 30046 maybe a processor of any suitable depth to perform the digital processingrequirements disclosed herein. For example, the processor 30046 a 32-bitprocessor, a 64-bit processor, a 128-bit processor, or the like.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The memory 30036 may include any component or collection of componentssuitable for storing data. For example, the memory 30036 may includevolatile memory and/or nonvolatile memory. The memory 30036 may includerandom-access memory (RAM), read-only memory (ROM), erasableprogrammable read-only memory (EPROM), (electrically erasableprogrammable read-only memory) EEPROM, flash memory, or the like.

The storage 30038 may include any component or collection of componentssuitable for storing large quantities of data. For example, storage30038 may include hard disk drives (HDD), solid state drives (SSD),network-attached storage (NAS), or the like. The storage 30038 mayinclude a database structure and/or a database management system (DBMS).

The display adapter 30040 may include any component or collection ofcomponents suitable for outputting the visual representation of a 3Dsimulation environment. For example, the display adapter 30040 mayinclude a graphics card, a display card, a graphics adapter, or thelike. The display adapter 30040 may be used to generates a feed ofoutput images to a display device, such as a display of the humaninterface device 30004. The display adapter 30040 may include a graphicsprocessing unit (GPU). The display adapter 30040 may include hardware torender a graphics pipeline, for example. The manipulation interfaceadapter 30042 may include any component or collection of componentssuitable for receiving manipulation information from the human interfacedevice and/or outputting feedback information to the human interfacedevice. For example, the manipulation interface adapter 30042 mayreceive motion tracking information from a virtual reality headset andin turn, manipulate the view being displayed to the user. For example,the manipulation interface adapter 30042 may receive control inputindicative of a user manipulating a surgical instrument and, in turn,output haptic feedback to the user's handheld device. For example, themanipulation interface adapter 30042 may receive control informationfrom a traditional desktop keyboard and mouse. The manipulationinterface adapter may include input/output hardware such as serialinput/output ports, parallel input/output ports, universal asynchronousreceiver transmitters (UARTs), discrete logic input/output pins,analog-to-digital converters, digital-to-analog converters, universalserial bus (USB) ports, USB-C ports, FireWire ports, High PerformanceParallel Interface (HIPPI), Thunderbolt port, Yapbus, Ethernet, GigabitEthernet, and/or any other suitable peripheral interface technology.

The surgical data system adapter 30044 may include any component orcollection of components suitable for communicating with the surgicaldata system 30008. The surgical data system adapter 30044 may includecommunications hardware to establish a physical channel between thesimulation device 30000 and the surgical data system 30008. For example,the surgical data system adapter 30044 may include a communication portsuch as, a USB port, USB-C ports, FireWire ports, HIPPI port,Thunderbolt port, Yapbus port, Ethernet port, Gigabit Ethernet port,and/or any other suitable peripheral interface. The surgical data systemadapter 30044 may include hardware, software, and/or a combinationthereof to establish a logical channel between the simulation device30000 and the surgical data system 30008 over the network adapter 30046and the network 30048.

The network adapter 30046 may include any component or collection ofcomponents suitable for communication over a network, such as network30048 for example. The network adapter 30046 may enable communicationover networks such as local area networks (LANs), wide area networks(WANs), and/or mobile networks. LAN technologies may include FiberDistributed Data Interface (FDDI), Copper Distributed Data Interface(CDDI), Ethernet/IEEE 802.3, Token Ring/IEEE 802.5, Wi-Fi/IEEE 802.11,and the like. WAN technologies may include, but are not limited to,point-to-point links, circuit-switching networks like IntegratedServices Digital Networks (ISDN) and variations thereon,packet-switching networks, and Digital Subscriber Lines (DSL). Themobile networks may include communication links based on one or more ofthe following mobile communication protocols: GSM/GPRS/EDGE (2G),UMTS/HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A),new radio (NR) or 5G, etc.

In an embodiment, the network adapter 30046 may include a wirelessnetwork adapter, such as a 5G network adapter. Such a 5G network adapter30046 may use a 5G New Radio (NR) transceiver to provide enhanced mobilebroadband (eMBB) with ultra-reliable and low latency communications(URLLC). Such a 5G network adapter 30046 may use wireless bands, such ashigher wireless bands like the 3.5 Ghz-7 Ghz and/or the 24 GHz-48 GHzbands. The network 30048 servicing such a 5G network adapter 30046 mayinclude a public wireless network, a semi-private (e.g., networkslicing-based) network, and/or a fully private wireless network.

FIG. 9 is a block diagram depicting an example surgical simulator humanuser interface device 30004. The human user interface device 30004 isdepicted with an example hardware architecture. For example, the humanuser interface device 30004 may include a processor 30050, a memory30052, a display subsystem 30054, and/or a manipulation subsystem 30056.

The processor 30050 may include computer processing unit, graphicsprocessing unit, any suitable microcontroller, microprocessor, fieldprogrammable gate array (FPGA), application specific integrated circuit(ASIC), or the like, and/or any combination thereof that is suitable forhandling the processing associated with displaying visual informationreceived from the simulation device 30000, processing manipulationinformation for sending to the simulation device, processing feedbackinformation received from the simulation device 30000, and the like. Theprocessor 30050 may include a microcontroller to interface with one ormore local sensors to sense control manipulation from the user and/or tointerface with one or more local actuators to provide feedback from theuser.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The memory 30036 may include any component or collection of componentssuitable for storing data. For example, the memory 30036 may includevolatile memory and/or nonvolatile memory. The memory 30036 may includerandom-access memory (RAM), read-only memory (ROM), erasableprogrammable read-only memory (EPROM), (electrically erasableprogrammable read-only memory) EEPROM, flash memory, or the like.

The display subsystem 30054 may include any component or collection ofcomponents suitable for displaying visual representations of a 3Dsimulation from the simulation device 30000 to a user. The displaysubsystem may include display hardware such as a monitor, a digitalprojector, a smart phone, a digital headset, a virtual reality headset,a stereoscopic display, a robotic surgery surgeon's console display, asurgical display unit, a surgical microscope, and the like.

The manipulation subsystem 30056 may include any component or collectionof components suitable for collecting manipulation controls from theuser to send to the simulation device 30000 and/or providing feedbackinformation, received from the simulation device 30000, to the user.Manipulation from the user may include any interface with sensors thatengage with the user, for example, engaging to indicate a user's intentin the simulation. For example, the interfaces may include keyboards,mice, joysticks, physical equipment that mimics the size, shape, andoperation of actual surgical instruments, virtual reality hand-heldcontrollers, smart gloves, motion sensing systems (such as hand trackingsystems, for example), a robotic surgery surgeon's console manipulatorsand/or controls, a physical unit that mimics the size, shape, andoperation of an actual robotic surgery surgeon's console manipulatorsand/or controls, and the like. For example, the interface may include apoint of view sensor, such as an accelerometer, in a headset to indicatea user's point of view within the simulation.

Feedback from the simulation device 30000 may include any interface withan actuator that provides sensory input to the user. For example, thefeedback may include haptic feedback, force feedback, temperaturefeedback, moisture feedback, audio feedback, olfactory feedback, and thelike. For example, a force feedback and/or haptic actuator in themanipulator of a robotic surgery surgeon's console may be used tosimulate the feedback the user would feel if operating such amanipulator in a live procedure. For example, a force feedback and/orhaptic actuator in a user device that mimics the size, shape, andoperation of actual surgical stapler may be used to simulate thefeedback the user would feel if operating such a device on live tissue,including force feedback when engaging the tissue and firing the staplerfor example.

FIG. 10 is a flow chart of an example surgical simulator operation. At30058, a simulation application may be loaded. For example, the coresimulation module 30016 may cause data associated with a particularapplication module 30010 to be loaded into memory 30036. The loaded datamay include instructions for the processor 30034 to operate a particularsimulation. The loaded data may include a procedural plan for thesimulation. For example, the procedural plan may be structured asdisclosed herein, for example with regard to FIGS. 11A-B. The loadeddata may include an initial state for the simulation.

At 30060, the simulation output may be determined and/or sent. Forexample, the simulation output may be determined and/or sent by thesimulation device 30000. Here, the core simulation module 30016 mayreference a current state of the simulation (e.g., an initial stateand/or a subsequent state). The core simulation module 30016 may engageone or more other modules to process the current state for output. Forexample, the core simulation module may engage any of the objectproperties module 30020, the texture module 30026, the applicationmodule 30010, the 3D graphics pipeline 30028, the interface module30012, and/or the surgical data system interface module 30014 to processthe current simulation state into information for output. Informationrelated to the output maybe processed and/or stored by the metricsextraction module 30030 and/or the session storage and management module30032, for example.

In a human-operated simulation session, for example, output informationmay be sent via the display adapter 30040 and/or the manipulationinterface adapter 30042 to the display subsystem 30054 and/or themanipulation subsystem 30056 of the human interface device 30004. In acomputer-controlled simulation session, for example, output informationmay be sent via the interface module 30012 to a surgeon agent 30006.Also for example, in a computer controlled simulation session, outputinformation may be sent (e.g., processed locally) at an applicationmodule 30010. In a session accessed via the surgical data system 30008,for example, output information may be sent by the surgical data systeminterface module 30014 via the surgical data system adapter 30044 and/orthe network adapter 30046.

At 30062, simulation input may be received and/or processed. Forexample, simulation input may be received and/or processed by thesimulation device 30000. Here, the core simulation module may engagewith the interface device, the surgical data system interface module,and/or the application module 30010 to receive control input.Information related to the input maybe processed and/or stored by themetrics extraction module 30030 and/or the session storage andmanagement module 30032, for example.

In a human-operated simulation session, for example, input informationmay be sent from a manipulation subsystem 30056 of the human interfacedevice 30004 and received via the manipulation interface adapter 30042.In a computer-controlled simulation session, for example, inputinformation may be sent from a surgeon agent 30006 and received via theinterface module 30012. Also for example, in a computer controlledsimulation session, input information may be received (e.g., processedlocally) at an application module 30010. In a session accessed via thesurgical data system 30008, for example, input information may bereceived via the surgical data system adapter 30044 and/or the networkadapter 30046 and initially handled by the surgical data systeminterface module 30014.

At 30064, a subsequent simulation state may be determined. For example,a subsequent simulation state may be determined from the currentsimulation state and/or the any received input. The core simulationmodule 30016 may engage one or more of the other modules of thesimulation device 30000 to determine the subsequent simulation state.For example, the code simulation module 30016 may engage the applicationmodule, the object properties module, the physics module, the physiologymodule, and the like. The subsequent simulation state may be determinedby operation of the processor 30034. Information related to the inputmaybe processed and/or stored by the metrics extraction module 30030and/or the session storage and management module 30032, for example.

At this stage, the process may loop to receiving input at 30060. Eachiteration of this flow may represent a corresponding time cycle in thesimulation. The framerate of the simulation may be set to a levelsuitable for the goal of the simulation and the processing capabilitiesof the surgical simulation device 30000. Lower framerates may enableprocessing that achieves a live human interaction simulation. Higherframerates may enable greater simulation fidelity. For example, whenoperating computer-controlled simulations, with a surgeon agent 30006for example, a higher framerate may be used, even if the higherframerate causes the processing time of the simulation to exceed thereal-world time it is simulating.

FIGS. 11A-B illustrate example surgical procedural plan data structuresfor use with a computer-implemented interactive surgical system and/or asurgical simulator. A surgical procedure plan may include informationthat outlines the staff, equipment, technique, and steps that may beused to perform a surgical procedure. For example, the procedure planmay include a staff manifest indicating what roles and/or what specifichealth care professionals are to be involved in the procedure. Theprocedure plan may include a listing of equipment, such as durablesurgical equipment, imaging equipment, instruments, consumables, etc.that may be used during the procedure. For example, the procedure planmay include a pick list for a surgical technician to use to assemble theappropriate tools and materials for the surgeon and the surgery whenprepping the operating theater. The procedure plan may includeinformation about the procedure's expected technique. For example, theprocedure plans for the same surgical goal may include different methodsof access, mobilization, inspection, tissue joining, wound closure, andthe like.

The procedure plan may reflect a surgeon's professional judgement withregard to an individual case. The procedure plan may reflect a surgeon'spreference for and/or experience with a particular technique. Theprocedure plan may map specific surgical tasks to roles and equipment.The procedure plan may provide an expected timeline for the procedure.

The procedure plan may include one or more decision points and/orbranches. Such decision points and/or branches may provide surgicalalternatives that are available for particular aspects of the procedure,where selection of one of the alternatives may be based on informationfrom the surgery itself. For example, the choice of one or morealternatives may be selected based on the particular planes of theparticular patient's anatomy, and the surgeon may select an alternativebased on her assessment of the patient's tissue during the live surgery.

The procedural plan may include one or more contingencies. These mayinclude information about unlikely but possible situations that mayarise during the live surgery. The contingencies may include one or moresurgical tasks that may be employed if the situation does occur. Thecontingencies may be used to ensure that adequate equipment, staff,and/or consumables are at the ready during the procedure.

The procedure plan may be recorded in one or more data structures. Aprocedure plan data structure may be used to record data about a futurelive surgery, about a completed live surgery, about a future simulatedsurgery, about a completed simulated surgery, and the like. A procedureplan data structure for live surgeries may be used by thecomputer-implemented interactive surgical system 100. For example, theprocedure plan data structure for live surgeries may be used by surgicalhub 106 to enhance situational awareness and/or the operational aspectsof the computer-implemented interactive surgical system 100. Theprocedure plan data structure for live surgeries may be used by thesurgical hub 106 to record discrete elements of the live surgery forstructured analysis.

A procedure plan data structure may be used by a simulation device30000. For example, the procedure plan data structure may be used by thesimulation device 30000 to establish a setting and/or one or moreobjectives for a simulation session. For example, the procedure plandata structure may be used by the simulation device 30000 to record thediscrete elements of the simulated surgery for structured analysis.

The procedure plan data structure may include any structure suitable forcapturing data elements related to the procedure. For example, theprocedure plan may be recorded in a tree-like data structure, such asthe one shown in FIG. HA, for example. Here, the root of the treestructure represents the core procedure data 30066. The core proceduredata 30066 may include information about the procedure as a whole, suchas procedure name, procedure code, patient name, date, time, and thelike. For a simulation, the core procedure data 30066 may includeinformation about simulation device, such as device ID, softwareversion, user, the simulation run settings, such as frame rate,resolution, connected user interface devices, and the like.

The procedure data may include leaves of the tree structure. The firstlevel of leaves may include data regarding the main aspects of theprocedure plan, such as the procedure setup 30068, one or more procedurestages 30070, one or more contingencies 30072, and the data regardingthe result of the procedure 30074.

The setup data 30068 may include information related to the preparationsand/or initial state of the procedure. For example, the setup data 30068may include elements such as staff manifest, staff roles and/or staffIDs, operating room ID, an equipment list, a room layout, an initialsurgical table position, a listing of instruments and/or consumables onprepared in the surgical field, any initial settings associated withequipment, pre-surgical imaging, patient record, etc. For a simulation,the setup data 30068 may include information related the simulatedenvironment, such as a record of the simulated anatomy, a record of thesimulated physiology, pre-surgical imaging, and the like.

The stage data 30070 may include data elements related to a majormilestone of the procedure. For example, a stage of a procedure mayinclude a milestone such as establishing access. The stage data 30070may include information related to the staff, equipment, technique, andsteps that may be used to perform the particular stage of the procedure.The stage data 30070 may include a stage ID.

The stage may be further detailed by one or more sub-leaves, such as oneor more surgical tasks 30076. The surgical task may represent a discretesurgical step within a given stage. For example, within the stage ofaccess, placing a trocar may be a surgical task. The surgical task data30076 may include a task ID. The surgical task data 30076 may includeinformation related to the particular task, such as the staff and/orsurgeon performing the task, the equipment to be used, the particulartechnique being applied, the patient vitals at the time the task isbeing performed, other environment information, and the list. Each taskmay be further detailed with goal data 30078, data related to ananatomy-instrument interaction 30080, and result data 30082. The goaldata 30078 may include information indicative of the relative success ofthe task performance. The goal data 30078 may include information aboutexpected task duration, acceptable performance specificity, efficiencymodality, avoidance of complications, and the like. The result data30082 may include information related to one or more goals. The resultdata 30082 may record the surgical performance (e.g., live and/orsimulated) relative to the goals.

The task data 30076 may include one or more elements ofanatomy-instrument interaction data 30080. The anatomy-instrumentinteraction data 30080 may represent a granular indication of surgicalperformance. The anatomy-instrument interaction data 30080 may representthe one or more specific activities used to perform the surgical task.The anatomy-instrument interaction data 30080 may represent theobservable behavior of the surgeon.

In an example, the anatomy-instrument interaction data 30080 may includethe specific positions, forces, angles, and the like being applied tothe anatomy by the surgeon. For example in a live surgery, data recordedfrom smart instruments by the surgical hub 106 may be captured asanatomy-instrument interaction data 30080. For example, a smart surgicalstapler in cooperation with other elements of the computer-implementedinteractive surgical system 100 may record stapler position, angle, tipforces, jaw forces, staple cartridge type, closing pressure, firingrate, and the like. In a simulated surgery, similar data elements may becaptured.

The contingency data 30072 may indicate any complications that may berelevant to the procedure. Each contingency data 30072 may include oneor more task data elements 30084 that address the appropriate responseto the particular complication. The contingency data 30072 may indicatedeviations from an original procedure plan. Also for example,contingency data may be cross-referenced to one or more tasks 30078and/or anatomy-instrument interactions 30080. For example, if a certainperformance in an anatomy-instrument interactions 30080 could lead to acomplication, the nature of that performance and a cross-reference tothe contingency may include in the result data 30082 associated withthat anatomy-instrument interactions 30080.

The result data 30074 may be indicative of the result of the procedure.Here overall metrics of the surgical performance may be stored, notes,actual and/or simulated patient recovery information, and/or patientoutcomes. For example, the result data 30074 may include efficiencyinformation, cost information, surgical duration, workload metrics,percentage of planned consumables used, and the like.

FIG. 11B illustrates a procedural plan data structure with the abovedisclosed elements, which further establishes structure of alternativesteps for completing a particular procedure, task, or activity. Asshown, the procedure represented by the procedure data 30086 may includetwo alternative setups, each indicated by respective setup data-a firstsetup data 30088, 30090 and a second setup data 30092. The first setupdata 30088, 30090 may include two alternative tasks 30094, 30096. Thesecond setup data 30092 may include one task 30098. In thisillustration, the procedure represented by procedure data 30086 may beaccomplished in three different ways. First via first setup 30088 andthe first task 30094. Second via the first setup 30090 and the secondtask 30096. And third via the second setup 30092 and its correspondingtask 30098.

Each path of the tree structure may represent a particular set ofalternative ways to perform the procedure. Such a structure may beuseful to aid the creation of a particular procedure plan for aparticular live and/or simulated surgery. Such a structure may be usefulto simulate many possible alternatives of a procedure to assess thedifferences in results.

FIG. 12 shows example surgical consequences management in a simulatedsurgical task.

At 30500, a simulation environment may be received. In examples, thesimulation environment may be used to simulate one or more surgicaltasks. The simulation environment may be defined by a set ofenvironmental parameters. For example, the set of environmentalparameters may define an anatomy and/or a physiology of a simulatedpatient. For example, the environmental parameters may define a colon(e.g., properties of a colon) and/or bodily structures surrounding thecolon. An example of an environmental parameter that defines a colon maybe colon friability. The environmental parameter may be set with anumeric value that, for example, that is representative of the colon'sfriability.

In examples, the environmental parameters may indicate a condition ofthe anatomy and/or physiology of the simulated patient. For example, theenvironmental parameters may indicate that the colon lacks adequateblood supply. For example, the environmental parameters may be set witha numeric value that is representative of the colon's blood supply. In acase where the colon lacks adequate blood supply, the numeric value maybe lower than an expected colon blood supply. In examples, the set ofenvironmental parameters may indicate a baseline anatomy and/orphysiology. In examples, the consequence environmental parameters mayindicate a modified anatomy, for example, that may be a result of asurgical interaction represented by the surgical interaction data.

In examples, defining the set of environmental parameters may involvethe object properties module and/or the physiology module, as describedwith respect to FIG. 7. The environmental parameters may be sent, forexample, by the simulator to a display module that may generate imagesto be used in the simulation environment. The images may represent theanatomy and/or physiology of the patient defined by the environmentalparameters and/or consequence environmental parameters as describedherein. Generating the images may involve the texture module and/or the3D graphics pipeline as described with respect to FIG. 7.

The surgical task(s) may be associated with a medical procedure. Forexample, a medical procedure may involve a surgeon performing one ormore surgical tasks to treat a patient's condition. In an example, themedical procedure may be a bariatric surgery. The surgical tasksassociated with bariatric surgery may be stomach access, stomachmobilization, and/or gastric transection. The simulator may be aware ofthe one or more surgical tasks that may be simulated. A user of thesimulator may perform each of the surgical tasks, for example, in orderto complete the medical procedure. The surgical task(s) may comprisesimilar characteristics as the task described with respect to FIGS. 11Aand/or 11B.

In examples, the surgical task(s) may be displayed to a user of thesimulator on a display screen. Information relevant to the surgicaltask(s), such as recommended medical instruments and/or a recommendedcourse of action, may be displayed on the display screen. The displayscreen may be included on the human interface device as described withrespect to FIG. 8.

The surgical task(s) may be associated with a medical procedure context.The medical procedure context may indicate a condition of the simulatedpatient's anatomy and/or physiology. In examples, the medical procedurecontext may be associated with the environmental parameters. Forexample, the medical procedure context may include a subset of theenvironmental parameters. For example, the medical procedure may be acolorectal surgery and the medical procedure context may include thenumeric value associated with colon friability as described herein. Inexamples, the simulator may generate the medical procedure context fromthe environmental parameters. In example, the medical procedure contextmay comprise tissue friability, tissue fragility, blood flow, tissueperfusion, allergic reaction, blood pressure, and/or heart rate.

The medical procedure context may be sent by the simulator to a surgicaldata system as described with respect to FIG. 7. In examples, themedical procedure and the medical procedure context may be sent to thesurgical data system as described with respect to FIG. 7. The surgicaldata system may use the medical procedure context, for example, whendetermining surgical interaction data to send to the simulator.

At 30505, surgical interaction data may be received. The surgicalinteraction data may be received from a surgical data system, asdescribed with respect to FIG. 7. In examples, the surgical interactiondata may be sent from the surgical data system to a simulator. Thesimulator may send a request message to the surgical data systemrequesting the surgical interaction data. The request message requestingthe surgical interaction data may comprise the medical procedure and themedical procedure context. In examples, the request message may be sentperiodically. The request message may be sent, for example, aftersignificant user interaction data is received by the simulator asdescribed herein.

The request message requesting the surgical interaction data may be sentbased on a surgical task that may be simulated. For example, thesimulator may be simulating a colorectal surgery. The colorectal surgerymay include the surgical task of mobilizing the colon. The simulator mayrequest surgical interaction data related to mobilizing the colon, forexample, if the simulator is simulating mobilizing the colon. Inexamples, surgical interaction data may be based on the medicalprocedure context as described herein. For example, the surgicalinteraction data may be based on a colon with low colon friability asindicated by the medical procedure context. The surgical data system mayfilter the surgical interaction data based on the medical procedurecontext and filtered surgical interaction data may be sent to thesimulator.

At 30510, user interaction data may be received. User interaction datamay be received by a simulator. The user interaction data may involvethe physics module as described with respect to FIG. 7. The userinteraction data may be based on one or more user actions within asimulation environment. For example, a simulator may be simulating acolorectal surgery. A user may ligate a simulated internal mammaryartery (IMA), for example, to perform mobilizing the colon surgicaltask. The user may use an input device as described with respect to FIG.14 to perform such actions in the simulation. In such a case, thephysics module may measure data related to the user ligating thesimulated IMA. For example, the simulator may measure the force exertedby the user on the input device as the user ligates the IMA.

At 30515, user interaction data received by the simulator may becompared against one or more triggering events. The surgical interactiondata described herein may comprise the triggering event(s). Thetriggering event(s) may be generated in a surgical data system asdescribed with respect to FIG. 7. In examples, the surgical data systemmay store historical data associated with real-life medical proceduresand real-life medical procedure contexts. The surgical data system maybe owned and/or operated by a medical facility, such as a hospital. Insuch a case, the historical data may be associated with the real-lifemedical procedures and medical procedure contexts previously performedby the staff of the medical facility.

The surgical data system may analyze the historical data and mayidentify correlation(s) between a surgeon's actions and environmentalevents. The surgical data system may set triggering event(s) andcorresponding consequence environment parameters based on thecorrelations. For example, the surgical data system may identify, basedon historical data, that when a surgeon exerts a certain amount of forceon an IMA, the IMA tends to tear. In such a case, the surgical datasystem may set a triggering event that corresponds to the amount offorce exerted on the IMA and may set consequence environment parameterscorresponding to the IMA tearing. The surgical data system may link thetriggering event to the consequence environment parameters. For example,the triggering event corresponding to the amount of force exerted on theIMA may be linked to the consequence environment parameterscorresponding to the IMA tearing.

On condition that the user interaction data matches the triggeringevent, the simulation environment may be modified based on theconsequence environment parameters. For example, one or more of thenumeric values of the environmental parameters may be updated with oneor more numeric values of the consequence environment parametersassociated with the triggering event that matched the user interactiondata. The modifications may involve the object properties module asdescribed with respect to FIG. 7. The updated values may be sent to adisplay module that may generate images corresponding to the consequenceenvironment parameters. The generated images may be used by thesimulator to depict the modified simulated environment to the user.

In examples, the triggering event may be a force exerted on an IMA andthe corresponding consequence environment parameters may indicate theIMA tearing. The simulator may receive user interaction data associatedwith a user exerting force on the IMA as the user attempts to mobilize acolon during the simulation. The simulator may compare the userinteraction data against the triggering event associated with the forceexerted on the IMA. The force exerted by the user may match thetriggering event. In such a case, the simulator may identify that theuser interaction data matches the triggering event and may update theenvironment parameters with the consequence environment parameterslinked to the triggering event. In such a case, the simulationenvironment may change to show a torn IMA.

In examples, the surgical interaction data may be filtered to filteredsurgical interaction data. The filtered surgical interaction data maycomprise filtered triggering event(s) and/or filtered consequenceenvironment parameters. The filtered surgical interaction data may befiltered by the simulator and/or by the surgical data system. Thesurgical data system may filter the surgical interaction data based onthe medical procedure and the medical procedure context. For example,the simulator may send to the surgical data system the medical procedurecurrently being simulated. The simulator may send to the surgical datasystem medical context associated with the medical procedure. Thesurgical data system may filter the triggering events and consequenceenvironment parameters stored on it, for example, to correspond to themedical procedure and the medical procedure context.

In examples, one or more probabilities associated with the triggeringevents may be determined. In examples, the surgical data system asdescribed with respect to FIG. 7 may determine the probabilities basedon the historical data described herein. For example, the surgical datasystem may analyze the historical data and determine that 20% of thetime when a surgeon exerts a certain force on the IMA, a tear in the IMAresults. The surgical interaction data received by the simulator maycomprise the probabilities along with the triggering events and theconsequence environment parameters. The probabilities may be linked tothe triggering events and consequence parameters. For example, theprobability 20% may be linked to the triggering event associated withthe force exerted to tear an IMA. In such a case, the simulator mayupdate the environment parameters with the consequence environmentparameters 20% of the time when the user interaction data matches thetriggering event.

In examples, a triggering threshold associated with one or moretriggering events may be determined. The simulator may determine thetriggering threshold based on the triggering events. In examples, thesurgical data system may determine the triggering threshold based on thetriggering events. The surgical interaction data may comprise thetriggering threshold. In examples, the surgical data system may send thesurgical interaction data, comprising the triggering threshold, to thesimulator, for example, after receiving a message from the simulatorrequesting the surgical interaction data.

The triggering threshold may be compared against the user interactiondata. In examples, the simulator may receive user interaction datarelated to a surgeon performing a simulated surgical task. The simulatormay compare the user interaction data against one or more triggeringthresholds, for example, which may have been received from a surgicaldata system as described herein. The simulator may determine that theuser interaction data has crossed the respective triggering threshold.In such a case, the simulator may modify the simulation environmentbased on the consequence environment parameters as described herein.

In examples, surgical interaction criteria based on one or more surgicaltasks may be determined. The surgical data system may determine thesurgical interaction criteria based on the historical data associatedwith a medical procedure and a medical procedure context. The surgicalinteraction data may comprise the surgical interaction criteria. Thesurgical interaction criteria may be linked to the triggering eventsand/or the consequence environment parameters.

The simulator may consider the surgical interaction criteria, forexample, when deciding whether to modify the simulated environment basedon the consequence environment parameters. In examples, the simulatormay determine that the user interaction data matches the triggeringevent. In such a case, the simulator may assess the simulation criteriato determine whether to modify the simulated environment based on theconsequence environment parameters. In examples, the simulationinteraction criteria may include the duration of time remaining in thesimulated medical procedure and/or simulated surgical task. For example,the simulator may modify the simulation environment based on theconsequence environment parameters if the time remaining is greater thanor equal to an X value. The simulator may not modify the simulationenvironment based on the consequence environment parameters if the timeremaining is less than the X value. The simulation criteria may includethe experience level of the user. For example, the simulator may modifythe simulation environment based on the consequence environmentparameters if the user has an expert experience level. The simulator maynot modify the simulation environment based on the consequenceenvironment parameters if the user has a novice experience level.

FIG. 13 shows example surgical consequences management in a simulatedsurgical task.

A simulator 30520 may comprise a simulator module 30540. The simulatormodule 30540 may be included in the core simulation module as describedwith respect to FIG. 7. In examples, the simulator module 30540 may beone or more of the applications modules as described with respect toFIG. 7.

The simulator module 30540 may comprise a simulator script 30545. Thesimulator script 30545 may include executable instructions that pertainto the simulator 30520. The simulator module 30540 may run the script30545, for example, when executing a simulation. In examples, thesimulator script 30545 may be independent from the simulator module30540.

The script 30545 may include one or more simulation parameter values.The simulation parameter values may be used to define a simulationenvironment. The simulation environment may be associated with a medicalprocedure. For example, a simulator 30520 may simulate a colorectalsurgery. In such a case, the simulation environment may include thecolon, stomach, rectum, and/or tissues surrounding the colon. In such acase, the simulation environment may include blood arteries involved incolorectal surgery, such as the IMA.

Simulation environment information may be received 30525. The simulationenvironment information may relate to a medical procedure and a medicalprocedure context. The simulator may request simulation environmentinformation based on the medical procedure and medical procedure contextit intends to simulate. For example, the simulator may intend tosimulate a colorectal surgery where scar tissue surrounds the colon. Thesimulator may send data related to this information to a remote databasein a request message. The database may send a response message withsimulation environment information related to a colorectal surgery wherescar tissue surrounds the colon. The simulation environment informationmay be set as script simulation parameter values, for example, whenreceived by the simulation. For example, one of the script simulationparameter values may be a numeric value for the quantity of scar tissue.The simulator may input the numeric value, for example, based on theresponse message sent from the database.

The simulator module 30540 may simulate a simulation environment 30555associated with a medical procedure and a medical procedure context. Thesimulator module 30540 may simulate a surgical task associated with themedical procedure. For example, the simulator module 30540 may simulatecolon mobilization for a colorectal surgery medical procedure. Thesimulator 30520 may display steps for a user to do in order to performthe colon mobilization surgical task.

The user may interact with the simulation. In examples, the user mayinteract with the simulation via the interface module as described withrespect to FIG. 7. In examples, the user may use a user interface deviceas described with respect to FIG. 8 to interact with the simulation. Thesimulator module may request user interaction data 30530 related to theuser interactions. For example, the user interface device may store theuser interaction data 30530 and may send the user interaction data 30530to the simulator module 30540, for example, as a response to the requestfor the user interaction data 30530. In examples, the user interfacedevice may periodically send the user interaction data 30530 to thesimulator module 30540. In examples, the surgical data system asdescribed in FIG. 7 may store and send the user interaction data 30530to the simulator module 30540. For example, the user interface devicemay send the user interaction data 30530 to the surgical data system.

Surgical consequence information 30535 may be received. In examples, thesurgical consequence information 30535 may be received from the surgicaldata system described herein. The surgical consequence information 30535may include the surgical interaction data described herein. For example,the surgical consequence information 30535 may include one or moretriggering events and one or more consequence environment parametersassociated with the triggering events. The surgical consequenceinformation 30535 may be received by the simulator 30520 and stored in asurgical consequence database. In such a case, the surgical consequenceinformation 30535 may be referred to as surgical consequence data 30550.The surgical consequence database may allow the simulator 30520 toaccess the surgical consequence data 30550, for example, if thesimulator 30520 wants to use the surgical consequence data 30550 in asimulation.

The simulator module 30540 may request the surgical consequence data30550. In such a case, the simulator 30520 may send the surgicalconsequence data 30550 to the simulator module 30540. The simulatormodule 30540 may compare 30560 the user interaction data 30530 againstthe surgical consequence data 30550. For example, the simulator module30540 may compare the user interaction data 30530 against the triggeringevents. The simulator module 30540 may assess whether the userinteraction data 30530 matches the triggering events as described withrespect to FIG. 12.

On a condition that the user interaction data 30530 does not match thetriggering event, the simulator module 30540 may not modify thesimulation environment and output an unmodified simulation environment30565 as described with respect to FIG. 12. On a condition that the userinteraction data 30530 does match the triggering event, the simulatormodule 30540 may modify the simulation environment based on the surgicalconsequence data 30550 and output a modified simulation environment30570 as described with respect to FIG. 12. For example, the simulatormodule 30540 may assess that the user interaction data 30530 matches thetriggering event. In such a case, the simulator module 30540 may send arequest for the consequence environment parameters. The request may besent to the surgical consequence database. The request may be sent tothe simulator 30520. The consequence environment parameters may be sentto the simulator module 30540, for example, in response to the request.In examples, the simulator 30520 may direct the surgical consequencedatabase to send the consequence environment parameters to the simulatormodule 30540.

The consequence environment parameters may be used to modify thesimulation environment and output a modified simulation environment30570. In examples, the simulation environment parameters may bereplaced with the consequence environment parameters to modify thesimulation environment. As shown in FIG. 13, the consequence environmentparameters may be used to modify a simulation environment to show asevered artery. For example, images may be generated, as described withrespect to FIG. 12, based on the consequence environment parameters. Theconsequence environment parameters may be sent to a display module thatmay be in communication with the simulator module 30540.

FIG. 14 shows a layout of the simulator design.

The simulator 30575 may include a simulation module 30605. Thesimulation module 30605 may be the simulation module 30605 as describedwith respect to FIG. 13. The simulation module 30605 may be a coresimulation module as described with respect to FIG. 7. In examples, thesimulation module 30605 may be one or more application modules asdescribed with respect to FIG. 7.

The simulation module 30605 may be in communication with a surgicalconsequence module 30610. In examples, the surgical consequence module30610 may be a component of the simulator 30575. In examples, thesurgical consequence module 30610 may be separate from the simulator30575. The surgical consequence module 30610 may be a component of thesurgical data system as described with respect to FIG. 7.

The surgical consequence module 30610 may analyze the triggering eventsand the consequence environment parameters. For example, the surgicalconsequence module 30610 may link the triggering events with respectiveconsequence environment parameters. The surgical consequence module30610 may comprise a database structure that stores the triggeringevents and the consequence environment parameters.

The simulation module 30605 may send a message to the surgicalconsequence module 30610 requesting the triggering events and therespective consequence environment parameters. The request message maybe associated with a query. The request message may comprise atriggering event. In examples, the request message list of triggeringevents and probabilities associated with each triggering event.

The request message may be defined by a surgical consequence applicationprogramming interface (API) 30615. For example, the surgical consequenceAPI 30615 may define the format of the request message. The format mayfacilitate communication between the simulation module 30605 and thesurgical consequence module 30610. In examples, the surgical consequenceAPI 30615 may analyze the request message to confirm that the requestmessage is in a correct format. In examples, the surgical consequenceAPI 30615 may transform the request message into a format.

The surgical consequence API 30615 may provide security to the databeing exchanged between the simulation module 30605 and the surgicalconsequence module 30610. In examples, the surgical consequence API30615 may allow an authentication key to be used in the communicationbetween the simulation module 30605 and the surgical consequence module30610. The authentication key may allow the surgical consequence API30615 to confirm the identity of the module, for example, which helps toprevent hacking attacks. For example, the simulation module 30605 maysend an authentication key when sending the request message to thesurgical consequence module 30610. The surgical consequence API 30610may check the authentication key and confirm that the simulationmodule's identity. The surgical consequence API 30615 may allow therequest message to proceed to the surgical consequence module 30610, forexample, after checking the authentication key.

The surgical consequence module 30610 may query the database for thetriggering event, for example, after receiving the request message fromthe simulation module 30605. The surgical consequence module 30610 maypinpoint the triggering event that matches the triggering event. Thesurgical consequence module 30610 may pinpoint the consequenceenvironment parameters linked to the triggering event. The surgicalconsequence module 30610 may send the triggering event and theconsequence environment parameters to the simulation module 30605. Inexamples, the surgical consequence module 30610 may send only theconsequence environment parameters to the simulation module 30605.

The simulator 30575 may include a display module 30595. The displaymodule 30595 may be a component of the simulator 30575. In examples, thedisplay module 30595 may be separate from the simulator 30575. Thedisplay module 30595 may be in communication with the three-dimensional(3D) graphics pipeline as described with respect to FIG. 7. In examples,the display module 30595 may be an application module as described withrespect to FIG. 7.

The display module 30595 may generate images associated with asimulation environment. For example, the simulator 30575 may intend tosimulate a colorectal surgery and may want the simulation environment torepresent the environment that the surgeon sees during a colorectalsurgery. To represent such an environment, the simulator 30575 may usethe display module 30595 to generate images related to colorectalsurgery. In examples, the display module 30595 may access a remotestorage, for example, where images related to medical procedure arestored. In examples, the display module 30595 may store the images in alocal database.

The physics module 30600 may measure user interaction data based on aninput device 30585. In examples, the input device 30585 may be a roboticcontroller. A user may move the robotic controller to perform surgicaltasks associated with the simulation. In examples, the roboticcontroller may resemble a medical instrument associated with thesurgical task. For example, the robotic controller may resemble aharmonic scalpel if a user is performing colon mobilization surgicaltask.

The input device 30585 may include a trigger. The trigger may allow auser apply simulated force with a simulated medical instrument in thesimulation. For example, during a simulation of a colon mobilizationsurgical task, the user may press the trigger of the input. Dataassociated with the pressing may be received by the user interactionmodule. The user interaction module may filter the pressing data. Thesimulator 30575 may use the filtered pressing data to simulate the forcewith the simulated medical instrument. For example, an environmentalparameter of the simulator 30575 may represent the force of thesimulated medical instrument. The simulator 30575 may input the filteredpressing data as a value of the environmental parameter.

The input device 30585 may include an actuator 30590. The actuator 30590may be a rotary actuator and/or a linear actuator. The actuator 30590may demonstrate to the user how to move the input device 30585 in orderto perform the surgical task. Data associated with the actuator'smovements may appear on the display 30580 for the user to see.

The simulator 30575 may be in communication with a display 30580. Thedisplay 30580 may be accessible to a user of the simulator 30575. Thesimulator 30575 may output the images generated by the display module30595 to the display 30580 for the user to see. The display module 30595may generate the images based on the user interaction data received fromthe input device 30585. The images may be outputted as a real-time livestream of the simulation to the display 30580. In such a case, the usermay be able to see the result of the user's action on the input device30585. For example, the user may press the trigger and the displaymodule 30595 may generate an image showing the simulated medicalinstrument firing an energy beam.

Dynamic and adaptive interactive simulations for improved real worldprocedure feel may be provided. Dynamic adaptive interactive simulationmay react to user's interactions. The adaptive simulation reaction maycomprise adjustable aspects of the procedure, for example, which mayautomatically adjust based on the interaction of the simulation user'schoices and/or reactions. The reactions may be a summary of real-worlddata sets that may be an aggregation of compiled data. The adjustableaspects may include regional differences, co-morbidities, medicationreaction, and/or other treatments to the simulated patient.

Automatic customization of simulators with patient specific informationmay be provided. The simulated anatomy may be customized with real worldand/or selected parameters, for example, to allow the simulation to bealigned with the patient the simulation is trying to mimic. This may behelpful in training, for example, as the trainer may selectcomplications or other issues the resident may encounter to broaden theresident's expertise. In the case of real-world surgeons simulating aprocedure plan for a real patient the customization of the simulationmay allow the simulation to be as close to the real patient interactionas the surgeon may get to understand access, patient issues and/orco-morbidities, etc.

Artificial Intelligence compilation and updating with regional, patientdemographic, or real-world surgical data may be provided.

The dynamic variables and aspects of the simulation may be updated by amachine learning algorithm that may be capable of adjusting thesimulation based on the reactions of the user as well as based on thereal-world data sets of previous surgeries. The real-world data sets ofprevious surgeries may be an aggregation of the procedure done by asurgeon, the surgeries from that facility and/or a compilation ofsurgeons using hubs within the same network.

The machine learning algorithm may be updated by data from a cloudand/or remote system, for example, which may be compiling bestpractices, regional data on surgeries, and/or worldwide outcomes andstep-of-use from any number of other facilities worldwide.

The reactions of the simulated patient may be adjusted based on theactions, approaches, issues, resolution, etc. of the user of thesimulation. The patient parameters may be adjusted based on chosenco-morbidities, regional differences, health status, and/or may beeffects by the machine learning based on the medications and/or othertreatments received by the simulated patient.

The real-world information may be derived from procedure outcomes, forexample, from the region, population etc. and/or may be interpolationand/or aggregation of sub-biomarker measures and outcomes.

The patient parameters may be adjusted based on input from wearables(e.g., physical activity, blood pressure, heard rate, for x weeks pre-opdrive probability of complications in simulation and watch out intra opindications). The patient parameters may be adjusted based on input fromnon-wearables application-based patient inputs such as medications takenand when the medications were taken, diet, exercise, and/or sleep.

Probability-driven local complications and anastomotic variation todisplay, select, able, and/or toggle able likely variances may beprovided. Statistical and/or medial risk probabilities may be used tocreate a realistic variation of reactions of the body, physiology,organs etc. to the user's actions and/or device usage. The probabilityreaction may produce a range of adaptations that may differ with eachre-use of the system. The range of adaptations may give the interactiona real-world air of interaction with the patient and/or surgery beingsimulated.

Simulator to create scenario of the patient response to known or unknownallergies/allergic reactions to medications and/or materials may beprovided. The simulator may be used to train and develop staff toidentify signals and/or reactions. The scenarios may be tailored to apatient based on patient characteristics and/or known allergies.

The predefined simulated patient co-morbidities, allergies, etc. may bedefined and the system may utilize these as context as the simulation isexecuted. For example, if a step, job, and/or medicant may have adiffering reaction due to the complicated patient variable, thesimulation may include that new modified reaction.

Reactive and predictive variable consequences based on choices may beprovided. Reactive variables of the consequences include one or more ofthe following: blood flow, perfusion, lobectomy (e.g., artery or vein inpulmonary artery in lung, which may fill up like a balloon), vessels orbroncus, access (e.g., how the anatomy presents the vessels, which maybe out front), different disease states may include differing outcomefor vein and artery, micro or macro tissue tension, or technique (e.g.,which may predict complication rate of the surgeon based on technique).Reactive variables may be linked. Reactive variables may be predictiveof adverse event risk based on simulation user action.

For example, indication of tissue fracture prediction based on specifictissue deflection may indicate a force. The force may cause a deflectionand fracture, which may lead to stress. The stress may cause deflection,for example, due to too high of a retraction force on the liver oranother heavy organ to access structures underneath.

Adaptive difficulty teaching and/or reactive simulations for procedurefamiliarity may be provided. The adaptable option and/or reaction maycomprise an aspect that allows a user to set the probability ofchallenges and/or define the type and intensity of the reaction to varythe simulator's response to user choices.

The change of difficulty may include the likelihood of lower probabilitybut higher risk aspects, adverse events, and/or complications beingutilized and allowing the user to overcome them. There may be arandomness aspect to the adaptability. For example, if a user ran thesame simulation twice and did the same set of steps and interactions,there may a be possibility that the reaction of the simulation maydiffer.

In examples, bleeding likelihood, tissue fragility and tissue tearing,radiation treatment reactions of tissue friability and the fragility ofthe tissue or its unusual thinness, ablation therapy, melted, tissue andits impact on coagulation, inflammation, etc. may be included in theadaption and/or reaction.

1. A device for managing surgical consequences in a simulated surgicaltask, the device comprising: a processor configured to: receive asimulation environment to simulate a surgical task defined by a set ofenvironmental parameters; receive surgical interaction data, wherein thesurgical interaction data comprises an indication of a triggering eventand an indication of consequence environmental parameters associatedwith the triggering event; receive user interaction data based on a useraction within the simulation environment and associated with thesurgical task; and on a condition that the received user interactiondata matches the triggering event, modify the simulation environmentbased on the consequence environmental parameters.
 2. The device ofclaim 1, wherein the surgical interaction data is associated with thesurgical task and the surgical task comprises a medical procedure and amedical procedure context.
 3. The device of claim 2, wherein thetriggering event and the consequence environmental parameters are basedon historical data associated with the medical procedure and the medicalprocedure context, wherein the historical data is based on previouslyperformed live procedures recorded by a surgical hub.
 4. The device ofclaim 3, wherein the historical data is local data associated with amedical facility, wherein the previously performed live procedures areperformed by medical staff associated with the medical facility, whereinthe surgical hub is controlled by the medical facility.
 5. The device ofclaim 1, wherein the processor is further configured to: determine aprobability that the triggering event occurs; determine an updatesimulation event based on the probability; and on a condition that theupdate simulation event occurs, modify the simulation environment basedon the consequence environmental parameters.
 6. The device of claim 2,wherein the processor is further configured to: filter the surgicalinteraction data to filtered surgical interaction data, wherein thefiltered surgical interaction data comprises an indication of a filteredtriggering event and an indication of filtered consequence environmentparameters, wherein the filtered triggering event is based on themedical procedure and the medical procedure context.
 7. The device ofclaim 1, wherein the processor is further configured to: determine atriggering threshold associated with the surgical interaction data; andon a condition that the user interaction data crosses the triggeringthreshold, modify the simulation environment based on the consequenceenvironmental parameters.
 8. The device of claim 2, wherein theprocessor is further configured to: determine surgical event interactioncriteria based on the surgical task; and on a condition that thereceived user interaction data satisfies the surgical event interactioncriteria, modify the simulation environment based on the consequenceenvironmental parameters.
 9. The device of claim 8, wherein the surgicalinteraction criteria is associated with the medical procedure and themedical procedure context.
 10. The device of claim 1, wherein the set ofenvironmental parameters indicates a baseline anatomy, wherein theconsequence environmental parameters indicate a modified anatomy that isa result of a surgical interaction represented by the surgicalinteraction data.
 11. The device of claim 2, wherein the medicalprocedure context comprises at least one of: tissue friability, tissuefragility, blood flow, tissue perfusion, allergic reactions, bloodpressure, or heart rate.
 12. A method, comprising: receiving asimulation environment to simulate a surgical task defined by a set ofenvironmental parameters; receiving surgical interaction data, whereinthe surgical interaction data comprises an indication of a triggeringevent and an indication of consequence environmental parametersassociated with the triggering event; receiving user interaction databased on a user action within the simulation environment and associatedwith the surgical task; and on a condition that the received userinteraction data matches the triggering event, modifying the simulationenvironment based on the consequence environmental parameters.
 13. Themethod of claim 12, wherein the surgical interaction data is associatedwith the surgical task and the surgical task comprises a medicalprocedure and a medical procedure context.
 14. The method of claim 13,wherein the triggering event and the consequence environmentalparameters are based on historical data associated with the medicalprocedure and the medical procedure context, wherein the historical datais based on previously performed live procedures recorded by a surgicalhub.
 15. The method of claim 14, wherein the historical data is localdata associated with a medical facility, wherein the previouslyperformed live procedures are performed by medical staff associated withthe medical facility, wherein the surgical hub is controlled by themedical facility.
 16. The method of claim 13, further comprising:filtering the surgical interaction data to filtered surgical interactiondata, wherein the filtered surgical interaction data comprises anindication of a filtered triggering event and an indication of filteredconsequence environment parameters, wherein the filtered triggeringevent is based on the medical procedure and the medical procedurecontext.
 17. The method of claim 13, further comprising: determiningsurgical event interaction criteria based on the surgical task; and on acondition that the received user interaction data satisfies the surgicalevent interaction criteria, modifying the simulation environment basedon the consequence environmental parameters.
 18. The method of claim 12,wherein the set of environmental parameters indicates a baselineanatomy, wherein the consequence environmental parameters indicate amodified anatomy that is a result of a surgical interaction representedby the surgical interaction data.
 19. The method of claim 13, whereinthe medical procedure context comprises at least one of: tissuefriability, tissue fragility, blood flow, tissue perfusion, allergicreactions, blood pressure, or heart rate.
 20. A system for managingsurgical consequences in a simulated surgical task, the systemcomprising: a processor configured to: receive a simulation environmentto simulate a surgical task defined by a set of environmentalparameters; receive surgical interaction data from a surgical datasystem, wherein the surgical interaction data comprises an indication ofa triggering event and an indication of consequence environmentalparameters associated with the triggering event, wherein the surgicaldata system stores the triggering event and the consequenceenvironmental parameters; receive user interaction data based on a useraction within the simulation environment and associated with thesurgical task; determine whether the received user interaction datamatches the triggering event; and modify, based on the determinationwhether the received user data matches the triggering event, thesimulation environment based on the consequence environmentalparameters.